...

• DE_instances T1(500-1K-2k). (MARC BIB) on Mobius environment with Data Export tests for 500, 1k, 2k. with Default instances export profile.

• DE_instances T2(Max). (MARC BIB) on Mobius environment with Data Export tests for a maximal number of instances on each member tenant(The exact number of instances is in the section “Errors / Additional information”). with Default instances export profile.

• DE_holdings T3(500-1K-2k).Holdings on Mobius environment with Data Export tests for 500, 1k, 2k. with Default holdings export profile.

• DE_holdings T4(Max). Holdings on Mobius environment with Data Export tests for a maximal number of instances on each member tenant(The exact number of instances is in the section “Errors / Additional information”) with Default holdings export profile.

• DE_authority T5(500-1K-2k). Authority on Mobius environment with Data Export tests for 500, 1k, 2k. with Default authority export profile.

...

DE_holdings T3(500-1K-2k) For 2000 records Holdings data export duration increased from 32 seconds for the first tenant to about 420 sec for the 22nd tenant, then the processing time was approximately from 420 to 470 sec for tenants 23 - 61.

DE_instances T2(Max) and DE_holdings T4(Max)

Recommendations & Jiras

• Restart the tests DE_instances T2(Max) and DE_holdings T4(Max) with different POOL_SIZE, and BATCH_SIZE parameters to check Data-export duration improvement. [MDEXP-688] Make configurable params for Data-Export(POOL_SIZE, BATCH_SIZE) - FOLIO Jira (atlassian.net)

• Restart Data-Import(Authority) test on mcpt to check how Kafka topics affect performance(found in the scope of data-preparation) [MODELINKS-213] Data-import process creating incorrect topics in Kafka - FOLIO Jira (atlassian.net)

...

DE_authority T5(500-1K-2k). The table below contains durations for Data-export for jobs with “Default authority export profile” on 61 tenants with 1 set of data(all of the instances on each tenant). 

Service CPU Utilization

DE_instances T1(500-1K-2k). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 13%.

...

Service CPU usage during the test was less than 20%, and small spikes were on mod-data-export with CPU usage 35% at the beginning of the text.

...

DE_authority T5(500-1K-2k).

Service CPU usage during the test was stable and correlated with the number of exported records. Mod-consortia service increases CPU usage for a few seconds from 20 to 30%.

...

Memory Utilization

DE_instances T1(500-1K-2k). During testing, the memory usage does not increase

...

DE_holdings T4(Max). During testing, the memory usage(MU) does not increase. Only mod-inventory change MU from 20 to 27 and mod-data-export from 40 to 90%

...

...

DE_authority T5(500-1K-2k). During testing, the memory usage does not increase

...

DB CPU Utilization

DE_instances T1(500-1K-2k). Except for the database spikes that started at 18:15 and repeated every 30 minutes. CPU usage before tets was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;

...

DE_holdings T4(Max). CPU usage before tets was about 2.5%, during all Data Export holdings, CPU usage increased up to 5.3%;

...

DE_authority T5(500-1K-2k). During 3 tests CPU varied within 4-5%.

...

DB Connections

DE_instances T1(500-1K-2k). Maximal number of DB connections was 1190 for 500 DE; 1270 for 1000DE and 1340 for 2000 DE.

...

DE_holdings T4(Max). The number of DB connections at the beginning of the test was about 1500 and evenly decreased to 850 at the end of the test

...

DE_authority T5(500-1K-2k). During 3 tests NUmber of DB connections varied from 700 to 1100

...

DB Load

DE_instances T1(500-1K-2k).

...

DE_holdings T3(500-1K-2k).

...

DE_holdings T4(Max).

...

DE_authority T5(500-1K-2k).

...

SQL queries

DE_instances T1(500-1K-2k).

...

DE_holdings T3(500-1K-2k).

...

DE_holdings T4(Max).

...

DE_authority T5(500-1K-2k).

...

Errors / Additional information

...

Methodology/Approach

Infrastructure

PTF -environment mcpt

• 10 m6i11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances1 database instance, writer/readerNameMemory GIBvCPUsmax

  Name Memory GIBv CPUsmax_connections

  db.r6g.xlarge32 GiB4 vCPUs27314xlarge 128 GiB; 16 vCPU

• MSK tenant

  • 4 2 m5.2xlarge brokers in 2 1 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

Table The table contains modules, memory, and CPU parameters