Versions Compared

Old Version 2

changes.mady.by.user Olga Kondratenko

Saved on

compared with

New Version Current

changes.mady.by.user Olga Kondratenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  1. In this workflow, we are checking the performance of Data Import for Cornell. This testing is done to mimic Cornell load with increased DB connections in mod-srs and mod-srm. 
    Jira Legacy
    serverSystem JiraJIRA
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-555

...

One of the reasons is the huge volume of the database: for example, table mod_source_record_storage.marc_indexers has 1028022472 1 028 022 472 records in the database.

Data import with the profile "EBSCO ebooks new and updated" could run successfully with database sizes: db.r6g.2xlarge, db.r6g.4xlarge, db.r6g.8xlarge. With increasing database size we can observe decreasing data import job duration.

...

On these graphs, we can observe gaps that correspond to DB issues with increased connections for mod-srs and mod-srm to 500.


Appendix


Infrastructure

Records count :

  • mod_source_record_storage.marc_indexers = 1028022472

PTF -environment cptf2 

  • 6 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance one writer only


  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
    • 2 partitions per DI topic


Module
cptf2-pvt		Task Def. Revision   Module Version Task CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSize   MaxMetaspaceSizeR/W split enabled
mod-data-import5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:2.7.11204818442561292384512false
mod-authtoken5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.13.021440115251292288128false
mod-inventory-update5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.0.12102489612876888128false
mod-configuration5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.12102489612876888128false
mod-inventory-storage5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.024096369020483076384512false
mod-source-record-storage18-22579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.1025600500020483500384512false
mod-inventory7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.622880259210241814384512false
mod-di-converter-storage6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.0.52102489612876888128false
mod-source-record-manager14-21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.425600500020483500384512false
nginx-edge10579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.1421024896128000false
mod-quick-marc5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.01228821761281664384512false
nginx-okapi10579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.1421024896128000false
okapi-b12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.13168414401024922384512false
pub-okapi10579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800false
mod-data-import-converter-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import-converter-storage:1.15.22102489612876888128false

...