Combined test for data export worker v2.0.3- (Nolana) 2022-12-01
Overview
Per - PERF-329Getting issue details... STATUS the following tests were performed using the data export workflow (bulk edits, eHoldings, circulation log, and edifacts) to determine what load can the exports run successfully together
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 AWS region (comparing to 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region for Nolana 2.1.0-SNAPSHOT.85)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
---|---|---|---|---|---|---|---|---|
mod-data-export-spring | 1.5.0 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
mod-data-export-worker | 2.0.3 | 6 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
mod-users | 19.0.0 | 1 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
mod-notes | 4.0.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
mod-inventory | 19.0.1 | 1 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.1 | 1 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
mod-orders | 12.5.0 | 1 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
mod-orders-storage | 13.4.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
mod-circulation | 23.3.0 | 1 | 2 | 1024 | 1024 | 896 | 128m | 768m |
mod-circulation-storage | 15.0.0 | 1 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
mod-agreements | 5.4.0 | 1 | 2 | 128 | 1592 | 1488 | 512m | 968m |
nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | - | - |
MG- Morning Glory release
Summary
Test report for combined mod-data-export-worker functionality 2022-12-01.
- 24 different concurrent jobs can be performed successfully in about 19 minutes (time to process the longest one - 2 jobs -10k item records).
- Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules.
- CPU - nginx-okapi, nod-configuration, and mod-orders-storage is spiking up to 250% at the start of tests. For mod-users-180%, mod-notes-130%, and mod-agreements-200%. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
- For all tests - RDS CPU utilization did not exceed 77%.
In general, Nolana is more stable compared to the Morning Glory release. All were tested with a high load. For - PERF-335Getting issue details... STATUS , it can be reproduced only if the load on mod-data-export-worker is high and has a solution - UIBULKED-182Getting issue details... STATUS for Nolana release.
Results
Test Runs (initial assumption of the data sets for combo testing)
Test Run | Edifact | eHoldings | Bulk-edit user barcodes | Bulk edit item barcodes | Bulk edit Holdings HRIDs | Bursar | Circulation log | Memory Behavior (Trend) | Comment | Time to process | Results |
#1 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 1 job -2500 records | 1 job -10k records | 1 job -10k records | DNR | 35K records | memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules. | 21 jobs total | About 18 min (time to process the longest one - job -10k item records) | 21 successful. 2 eHolding jobs were scheduled for 1 min 45 sec. |
#2 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 2 jobs -2500 records each | 2 jobs -10k records each | 2 jobs -10k records each | DNR | 35K records | 24 jobs total | About 19 min (time to process the longest one - 2 jobs -10k item records) | ||
#3 | 10 jobs 1K orders each | 7 jobs 9,194 titles each | 2 jobs -2500 records each | 2 jobs -10k records each | 2 jobs -10k records each | DNR | 35K records | 24 jobs total | About 19 min (time to process the longest one - 2 jobs -10k item records) | ||
#4 | 1 job -2500 records | 1 job -10k records | 1 job -10k records | About 18 min (time to process the longest one - job -10k item records) | all successful. 1 eHolding and 2 Edifact jobs were scheduled for 1 min 45 sec. |
Test results
Test Run | Edifact | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | eHoldings | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Bulk-edit user barcodes | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Bulk edit item barcodes | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Comment |
#1 | 10 jobs 1K orders each | 1 m 20 s | 1 min 40 s | success | 7 jobs 9,194 titles each | 4 m 55s | ±10 min | success | 1 job -2500 records | ±2 min | 3 min 45 sec | success | 1 job -10k records | 17 min 30 sec | 26-27 min | success | |
#2 | 10 jobs 1K orders each | 2 m 20 s | 1 min 50 s | success | 7 jobs 9,194 titles each | 5 m 35 s | ±12 min | success | 2 jobs -2500 records each | 2min 20 sec | 2 min 10 sec for a successful job | success | 2 jobs -10k records each | ±19 min | 28 min 30 sec | success | |
#3 | 10 jobs 1K orders each | 2 m 5 s | 1 min 37 s | success | 7 jobs 9,194 titles each | 5 m 31 s | ±12 min | success | 2 jobs -2500 records each | 2min 20 sec | 2 min 50 sec for a successful job | success | 2 jobs -10k records each | ±19 min | 28 min 30 sec | success | |
#4 | 1 job -2500 records | 2 min 15 sec | 3 min 45 sec | success | 1 job -10k records | 18 min | 26-27 min | success |
Test Run | Bursar | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Circulation log | Time to process v2.0.3 | Results | Bulk edit Holdings HRIDs | Time to process v2.0.3 | Time to process Nolana's snapshot | Results | Comment |
#1 | DNR | DNR | DNR | 35K records | 5 m 15s | success | 1 job -10k records | ±7 min | - | success | ||
#2 | DNR | DNR | DNR | 35K records | 5 m 38 s | success | 2 jobs -10k records each | ±8 min | - | success | ||
#3 | DNR | DNR | DNR | 35K records | 5 m 3 s | success | 2 jobs -10k records each | ±8 min | - | success | ||
#4 | 1 job -10k records | 7 min 30 sec | - | success |
* - average job processing time for the number of records specified ( upload identifiers + editing)
Resource Usage
RDS CPU utilization
Test Run #1
Memory Trends of combined test:
- Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 49%. No leak suspect was found in the heap dump. Stable for all other modules.
CPU usage of combined test:
Instance CPU usage of combined test:
RDS CPU usage of combined test:
Additional information:
For 12 concurrent bulk edit jobs (4- 10k Items records, 4 - 2,5k users records, 4- 10k holdings records) all items and holdings jobs FAILED with response 500 or response 502
Example of error details:
[500 Internal Server Error] during [GET] to [http://inventory/items?query=barcode%3D%3D%22H79118086%22&limit=1] [InventoryClient#getItemByQuery(String,long)]: [Holdings request (id==(332c88e0-e712-4cec-a3a4-f7a620c4f2e7)) failed 502: <html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.18.0</center> </body> </html> ] (InternalServerError)
[502 Bad Gateway] during [GET] to [http://inventory/items?query=barcode%3D%3D%22H97502733%22&limit=1] [InventoryClient#getItemByQuery(String,long)]: [<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.18.0</center> </body> </html> ] (BadGateway)
ncp3/mod-inventory/e52839606c944c82b93fc1154d7a3a8b 13:37:01 [] [] [] [] ERROR Items Failed to retrieve bound-with parts, status code: 502
13:37:01 [] [] [] [] INFO ? 108095/bulk-edit;025851/instance-storage RES 502 2937us mod-inventory-storage-25.0.1 http://mod-inventory-storage-b.ncp3.folio-eis.us-east-1:8051/mod-inventory-storage/instance-storage/instances/35793b16-c6a1-48f7-9c2e-32670133dbbf