Combined test for data export worker v2.0.3- (Nolana) 2022-12-01
Overview
Per https://folio-org.atlassian.net/browse/PERF-329 the following tests were performed using the data export workflow (bulk edits, eHoldings, circulation log, and edifacts) to determine what load can the exports run successfully together
Infrastructure
PTF -environment
9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 AWS region (comparing to 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region for Nolana 2.1.0-SNAPSHOT.85)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-data-export-spring | 1.5.0 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
mod-data-export-worker | 2.0.3 | 6 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
mod-users | 19.0.0 | 1 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
mod-notes | 4.0.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
mod-inventory | 19.0.1 | 1 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.1 | 1 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
mod-orders | 12.5.0 | 1 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
mod-orders-storage | 13.4.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
mod-circulation | 23.3.0 | 1 | 2 | 1024 | 1024 | 896 | 128m | 768m |
mod-circulation-storage | 15.0.0 | 1 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
mod-agreements | 5.4.0 | 1 | 2 | 128 | 1592 | 1488 | 512m | 968m |
nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | - | - |
MG- Morning Glory release
Summary
Test report for combined mod-data-export-worker functionality 2022-12-01.
24 different concurrent jobs can be performed successfully in about 19 minutes (time to process the longest one - 2 jobs -10k item records).
Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules.
CPU - nginx-okapi, nod-configuration, and mod-orders-storage is spiking up to 250% at the start of tests. For mod-users-180%, mod-notes-130%, and mod-agreements-200%. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
For all tests - RDS CPU utilization did not exceed 77%.
In general, Nolana is more stable compared to the Morning Glory release. All were tested with a high load. Forhttps://folio-org.atlassian.net/browse/PERF-335, it can be reproduced only if the load on mod-data-export-worker is high and has a solutionhttps://folio-org.atlassian.net/browse/UIBULKED-182 for Nolana release.
Results
Test Runs (initial assumption of the data sets for combo testing)
Test Run | Edifact | eHoldings | Bulk-edit user barcodes | Bulk edit item barcodes | Bulk edit Holdings HRIDs | Bursar | Circulation log | Memory Behavior (Trend) | Comment | Time to process | Results |
#1 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 1 job -2500 records | 1 job -10k records | 1 job -10k records | DNR | 35K records |
| 21 jobs total | About 18 min (time to process the longest one - job -10k item records) | 21 successful. 2 eHolding jobs were scheduled for 1 min 45 sec. |
#2 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 2 jobs -2500 records each | 2 jobs -10k records each | 2 jobs -10k records each | DNR | 35K records | 24 jobs total | About 19 min (time to process the longest one - 2 jobs -10k item records) | 24 successful, 2 bulk edits scheduled with | |
#3 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 2 jobs -2500 records each | 2 jobs -10k records each | 2 jobs -10k records each | DNR | 35K records | 24 jobs total | About 19 min (time to process the longest one - 2 jobs -10k item records) | 24 successful, 2 bulk edits scheduled with | |
#4 |
|
| 1 job -2500 records | 1 job -10k records | 1 job -10k records |
|
|
| About 18 min (time to process the longest one - job -10k item records) | all successful. 1 eHolding and 2 Edifact jobs were scheduled for 1 min 45 sec. |
Test results
Test Run | Edifact | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | eHoldings | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Bulk-edit user barcodes | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Bulk edit item barcodes | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Comment |
#1 | 10 jobs 1K orders each | 1 m 20 s | 1 min 40 s | success | 7 jobs 9,194 titles each | 4 m 55s | ±10 min | success | 1 job -2500 records | ±2 min | 3 min 45 sec | success | 1 job -10k records | 17 min 30 sec | 26-27 min | success |
all successful, bulk edits scheduled with |
#2 | 10 jobs 1K orders each | 2 m 20 s | 1 min 50 s | success | 7 jobs 9,194 titles each | 5 m 35 s | ±12 min | success | 2 jobs -2500 records each | 2min 20 sec | 2 min 10 sec for a successful job | success | 2 jobs -10k records each | ±19 min | 28 min 30 sec | success | |
#3 | 10 jobs 1K orders each | 2 m 5 s | 1 min 37 s | success | 7 jobs 9,194 titles each | 5 m 31 s | ±12 min | success | 2 jobs -2500 records each | 2min 20 sec | 2 min 50 sec for a successful job | success | 2 jobs -10k records each | ±19 min | 28 min 30 sec | success | |
#4 |
|
|
|
|
|
|
|
| 1 job -2500 records | 2 min 15 sec | 3 min 45 sec | success | 1 job -10k records | 18 min | 26-27 min | success |
Test Run | Bursar | Time to process v2.0.3 |