Combined test for data export worker v2.0.3- (Nolana) 2022-12-01
- Olga Kondratenko
- Roman_Fedynyshyn
Overview
Per PERF-329 - Getting issue details... STATUS the following tests were performed using the data export workflow (bulk edits, eHoldings, circulation log, and edifacts) to determine what load can the exports run successfully together
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 AWS region (comparing to 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region for Nolana 2.1.0-SNAPSHOT.85)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-data-export-spring | 1.5.0 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
| mod-data-export-worker | 2.0.3 | 6 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
| mod-users | 19.0.0 | 1 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
| mod-notes | 4.0.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
| mod-inventory | 19.0.1 | 1 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.1 | 1 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
| okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
| mod-orders | 12.5.0 | 1 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
| mod-orders-storage | 13.4.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-circulation | 23.3.0 | 1 | 2 | 1024 | 1024 | 896 | 128m | 768m |
| mod-circulation-storage | 15.0.0 | 1 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
| mod-agreements | 5.4.0 | 1 | 2 | 128 | 1592 | 1488 | 512m | 968m |
| nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | - | - |
MG- Morning Glory release
Summary
Test report for combined mod-data-export-worker functionality 2022-12-01.
- 24 different concurrent jobs can be performed successfully in about 19 minutes (time to process the longest one - 2 jobs -10k item records).
- Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules.
- CPU - nginx-okapi, nod-configuration, and mod-orders-storage is spiking up to 250% at the start of tests. For mod-users-180%, mod-notes-130%, and mod-agreements-200%. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
- For all tests - RDS CPU utilization did not exceed 77%.
In general, Nolana is more stable compared to the Morning Glory release. All were tested with a high load. For PERF-335 - Getting issue details... STATUS , it can be reproduced only if the load on mod-data-export-worker is high and has a solution UIBULKED-182 - Getting issue details... STATUS for Nolana release.