Combined test for data export worker - (Nolana) 2022-11-08
Overview
Per https://folio-org.atlassian.net/browse/PERF-315 the following tests were performed using the data export workflow (bulk edits, eHoldings, and edifacts) to determine what load can the exports run successfully together
Infrastructure
PTF -environment
9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfka-1
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=truec
log.retention.minutes=480
default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-data-export-spring | 1.5.0-SNAPSHOT.58 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
mod-data-export-worker | 2.1.0-SNAPSHOT.85 | 2 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
mod-users | 19.0.0-SNAPSHOT.573 | 4 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
mod-notes | 4.0.0-SNAPSHOT.237 | 2 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
mod-inventory | 19.0.0-SNAPSHOT.383 | 2 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.0-SNAPSHOT.631 | 5 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
okapi | 4.14.4 | 2 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
mod-orders | mod-orders-12.5.0-SNAPSHOT.229 | 3 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
mod-orders-storage | mod-orders-storage-13.4.0-SNAPSHOT.154 | 2 | 2 | 128 | 1024 | 896 | 128m | 768m |
mod-circulation | mod-circulation-23.3.0-SNAPSHOT.544 | 3 | 2 | 1024 | 1024 | 896 | 128m | 768m |
mod-circulation-storage | mod-circulation-storage-15.1.0-SNAPSHOT.296 | 6 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
mod-agreements | 5.4.0-SNAPSHOT.104 | 2 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Summary
Test report for combined mod-data-export-worker functionality 2022-11-08.
Up to 10 different concurrent jobs can be performed successfully. Compared to individual bulk edit runs -10k records per user, 5 users simultaneously (50k records total) can be uploaded and updated in about 30 min which is successful. Bulk Edits with more than 10 concurrent jobs can fail with
MODEXPW-306: [Bulk edit] Index 798 out of bounds for length 13(ArrayIndexOutOfBoundsException)Closed or
Memory trend: memory usage increases for mod-data-export-worker by 1% (from 109% to 110%). Stable for all other modules.
CPU - nginx-okapi is spiking up to 170% at the start of tests. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
For all tests - RDS CPU utilization did not exceed 80%.
In general, Nolana is more stable compared to the Morning Glory release. All were tested with a higher load and
Results
Test Runs (initial assumption of the data sets for combo testing)
Bulk edit files are located at https://github.com/folio-org/perf-testing/tree/Bulk-edit/workflows-scripts/Bulk-edit
Test Run | Edifact | eHoldings | Bulk-edit user barcodes | Bulk edit item barcodes | Bulk edit Holdings HRIDs | Bursar | Circulation log | Memory Behavior (Trend) | Comment | Time to process | Results |
#1 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records |
| 28 jobs total | About 27 min (time to process the longest one - 5 jobs -10k item records) |
|
#2 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records | 28 jobs total | About 27 min (time to process the longest one - 5 jobs -10k item records) | 24 successful, 4 failed | |
#3 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records | 28 jobs total | About 29 min (time to process the longest one - 5 jobs -10k item records) | 24 successful, 4 failed | |
|
|
|
|
|
|
|
|
|
|
|
Test results
Test Run | Edifact | Time to process | Results | eHoldings | Time to process | Results | Bulk-edit user barcodes | Time to process | Results | Bulk edit item barcodes | Time to process | Results | Comment |
#1 | 10 jobs 1K orders each | 1 min 40 s | success | 7 jobs 9,194 titles each | ±10 min | success | 5 jobs -2500 records each | 3 min 45 sec | success | 5 jobs -10k records each | 26-27 min | success |
Sometimes Bulk edit jobs can be failed with |
#2 | 10 jobs 1K orders each | 1 min 50 s | success | 7 jobs 9,194 titles each | ±12 min | success | 5 jobs -2500 records each | 2 min 10 sec for a successful job | 1 Successful - 4 failed
| 5 jobs -10k records each | 27 min | success | |
#3 | 10 jobs 1K orders each | 1 min 37 s | success | 7 jobs 9,194 titles each | ±12 min | success | 5 jobs -2500 records each | 2 min 50 sec for a successful job | 1 Successful - 4 failed
| 5 jobs -10k records each | 28 min 30 sec | success |
Test Run | Bursar | Time to process | Results | Circulation log | Time to process | Results | Bulk edit Holdings HRIDs | Time to process | Results | Comment |
#1 | DNR | DNR | DNR | 35K records | 8 min 52 s | success |
|
|
|
|
#2 | DNR | DNR | DNR | 35K records | ±8 min | success |
|
|
|
|
#3 | DNR | DNR | DNR | 35K records | 7 min 50 s | success |
|
|
|
|
* - average job processing time for the number of records specified ( upload identifiers + editing)
Note- To manipulate with number of records to export in Circulation log workflow we just change dates in from-to section.
Resource Usage
Note: Between test #2 and test #3 we have restarted mod-data-export-worker manually.
RDS CPU utilization