Combined test for data export worker - (Nolana) 2022-11-08
- Olga Kondratenko
- Roman_Fedynyshyn
- Martin Tran
Overview
Per PERF-315 - Getting issue details... STATUS the following tests were performed using the data export workflow (bulk edits, eHoldings, and edifacts) to determine what load can the exports run successfully together
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfka-1
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-data-export-spring | 1.5.0-SNAPSHOT.58 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
| mod-data-export-worker | 2.1.0-SNAPSHOT.85 | 2 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
| mod-users | 19.0.0-SNAPSHOT.573 | 4 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
| mod-notes | 4.0.0-SNAPSHOT.237 | 2 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
| mod-inventory | 19.0.0-SNAPSHOT.383 | 2 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.0-SNAPSHOT.631 | 5 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
| okapi | 4.14.4 | 2 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
| mod-orders | mod-orders-12.5.0-SNAPSHOT.229 | 3 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
| mod-orders-storage | mod-orders-storage-13.4.0-SNAPSHOT.154 | 2 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-circulation | mod-circulation-23.3.0-SNAPSHOT.544 | 3 | 2 | 1024 | 1024 | 896 | 128m | 768m |
| mod-circulation-storage | mod-circulation-storage-15.1.0-SNAPSHOT.296 | 6 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
| mod-agreements | 5.4.0-SNAPSHOT.104 | 2 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Summary
Test report for combined mod-data-export-worker functionality 2022-11-08.
- Up to 10 different concurrent jobs can be performed successfully. Compared to individual bulk edit runs -10k records per user, 5 users simultaneously (50k records total) can be uploaded and updated in about 30 min which is successful. Bulk Edits with more than 10 concurrent jobs can fail with MODEXPW-306 - Getting issue details... STATUS or PERF-335 - Getting issue details... STATUS
- Memory trend: memory usage increases for mod-data-export-worker by 1% (from 109% to 110%). Stable for all other modules.
- CPU - nginx-okapi is spiking up to 170% at the start of tests. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
- For all tests - RDS CPU utilization did not exceed 80%.
In general, Nolana is more stable compared to the Morning Glory release. All were tested with a higher load and MODEXPW-306 - Getting issue details... STATUS occurs very rarely. As for PERF-335 - Getting issue details... STATUS , it can be reproduced only if mod-data-export-worker is overloaded and will be solved in UI.
Results
Test Runs (initial assumption of the data sets for combo testing)
Bulk edit files are located at https://github.com/folio-org/perf-testing/tree/Bulk-edit/workflows-scripts/Bulk-edit
Test Run | Edifact | eHoldings | Bulk-edit user barcodes | Bulk edit item barcodes | Bulk edit Holdings HRIDs | Bursar | Circulation log | Memory Behavior (Trend) | Comment | Time to process | Results |
#1 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records | memory usage increases for mod-data-export-worker during the test by 2% up to 111%. Stable for all other modules. | 28 jobs total | About 27 min (time to process the longest one - 5 jobs -10k item records) | |
#2 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records | 28 jobs total | About 27 min (time to process the longest one - 5 jobs -10k item records) | 24 successful, 4 failed | |
#3 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 5 jobs -2500 records each | 5 jobs -10k records each | - | DNR | 35K records | 28 jobs total | About 29 min (time to process the longest one - 5 jobs -10k item records) | 24 successful, 4 failed | |
Test results
Test Run | Edifact | Time to process | Results | eHoldings | Time to process | Results | Bulk-edit user barcodes | Time to process | Results | Bulk edit item barcodes | Time to process | Results | Comment |
| #1 | 10 jobs 1K orders each | 1 min 40 s | success | 7 jobs 9,194 titles each | ±10 min | success | 5 jobs -2500 records each | 3 min 45 sec | success | 5 jobs -10k records each | 26-27 min | success | Sometimes Bulk edit jobs can be failed with MODEXPW-306 - Getting issue details... STATUS or PERF-335 - Getting issue details... STATUS |
| #2 | 10 jobs 1K orders each | 1 min 50 s | success | 7 jobs 9,194 titles each | ±12 min | success | 5 jobs -2500 records each | 2 min 10 sec for a successful job | 1 Successful - 4 failed PERF-335 - Getting issue details... STATUS | 5 jobs -10k records each | 27 min | success | |
| #3 | 10 jobs 1K orders each | 1 min 37 s | success | 7 jobs 9,194 titles each | ±12 min | success | 5 jobs -2500 records each | 2 min 50 sec for a successful job | 1 Successful - 4 failed PERF-335 - Getting issue details... STATUS | 5 jobs -10k records each | 28 min 30 sec | success |
Test Run | Bursar | Time to process | Results | Circulation log | Time to process | Results | Bulk edit Holdings HRIDs | Time to process | Results | Comment |
| #1 | DNR | DNR | DNR | 35K records | 8 min 52 s | success | ||||
| #2 | DNR | DNR | DNR | 35K records | ±8 min | success | ||||
| #3 | DNR | DNR | DNR | 35K records | 7 min 50 s | success |
* - average job processing time for the number of records specified ( upload identifiers + editing)
Note- To manipulate with number of records to export in Circulation log workflow we just change dates in from-to section.
Resource Usage
Note: Between test #2 and test #3 we have restarted mod-data-export-worker manually.
RDS CPU utilization
Test Run #1
Memory Trends of combined test:
- Memory trend: memory usage increases for mod-data-export-worker by 1% (from 109% to 110%). Stable for all other modules.
CPU usage of combined test:
Instance CPU usage of combined test:
RDS CPU usage of combined test:
Concurrent Bulk Edit Items App + Bulk Edit Users App -30 jobs total
Test Run | Bulk-edit user barcodes | Bulk edit item barcodes | Memory Behavior (Trend) | Comment | Time to process | Results |
#1 | 5 jobs-5k records each+ 5 jobs -2500 records each | 5 jobs-5k records each+ 5 jobs -10k records each+ 5 jobs-1k records each+ 5 jobs-100 records each | memory usage increases for mod-data-export-worker during the test by 2% up to 109%. Stable for all other modules. | 30 jobs total | About 29 min (time to process the longest one - 5 jobs -10k item records) | 28 successful, 2 failed |
Test results
Test Run | Bulk-edit user barcodes | Time to process | Results | Bulk edit item barcodes | Time to process | Results | Comment |
| #2 | 5 jobs -2500 records each | 7 min 30 sec * | 4 Successful -1 failed MODEXPW-306 - Getting issue details... STATUS | 5 jobs -100 records each | 1 min 30 sec * | Successful | Sometimes jobs can be failed with PERF-335 - Getting issue details... STATUS |
| 5 jobs -5k records each | 14 min * | 4 Successful -1 failed MODEXPW-306 - Getting issue details... STATUS | 5 jobs -1k records each | 5 min 30 sec * | Successful | ||
| 5 jobs -5k records each | 16-18 min | Successful | |||||
| 5 jobs -10k records each | 27-29 min | Successful |
For 30 parallel bulk edits 28 out of 30 -successful
- Memory trend: memory usage increases for mod-data-export-worker by 2% and up to 109%. Stable for all other modules.
CPU usage
RDS CPU Utilization did not exceed 27%
