Bulk Edit Items App report [Nolana] 27/10/2022
- Olga Kondratenko
Overview
Bulk Edit - Establish a performance baseline for user status bulk updates.
- How long does it take to export 100, 1000, 2500, 10k, and 100K records?
- Use it for up to 5 concurrent users.
- Look for a memory trend and CPU usage
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfka-1
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
| # | modules | task definition | running tasks | CPU | memory | memoryReservation | maxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-data-export-spring | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) | |
| mod-data-export-worker | 3 | 2 | 1024 | 3072 | 2600 | 612 | 2128 (1536 in MG) | |
| mod-users | 4 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 | |
| mod-notes | 2 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) | |
| mod-inventory | 2 | 2 | 1024 | 2880 | 2592 | 512m | 1814m | |
mod-inventory-storage | 5 | 2 | 1024 | 2208 | 1952 | 512m | 1440m | |
| okapi | 2 | 3 | 1024 | 1684 | 1440 | 512m | 922m | |
| mod-orders | 3 | 2 | 1024 | 2048 | 1440 | 512m | 896m | |
| mod-orders-storage | 2 | 2 | 128 | 1024 | 896 | 128m | 768m | |
| mod-circulation | 3 | 2 | 1024 | 1024 | 896 | 128m | 768m | |
| mod-circulation-storage | 6 | 2 | 1024 | 1536 | 1440 | 512m | 896m | |
| mod-agreements | 2 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Software Versions
- mod-data-export-worker-1.5.0-SNAPSHOT.76
- mod-data-export-spring-1.5.0-SNAPSHOT.58
- mod-agreements-5.4.0-SNAPSHOT.104
- mod-notes-4.0.0-SNAPSHOT.237
- mod-users-19.0.0-SNAPSHOT.573
- mod-inventory-19.0.0-SNAPSHOT.383
- mod-inventory-storage-25.0.0-SNAPSHOT.631
- okapi-4.14.4
Summary
Test report for Bulk Edits items-app functionality 2022-10-27.
- 10k records per user, 5 users simultaneously (50k records total) can be uploaded in about 10 min 45 seconds, and edited in 17 minutes (about 28 min total).
- The files with identifiers should be strictly determined.
- The memory of mod-data-export-worker increased from 88% at the start of the tests and was about 102% at the end. Each step of memory increase is related to CPU spikes at the beginning of the records update. mod-inventory-storage memory increased from 80% to 94% (the possible reason is stepped increasing while another process starts on the server (indexing and EDIFACT)). mod-inventory memory increased by 2% (from 60% to 62% and become stable) after 2 of 3 50k records updating jobs were finished and only 1 job remaining. All other modules' memory usage was stable.
- CPU - for 5 parallel jobs - for mod-data-export-worker are spiking up to 120% at the start of records updating( in general usage was about 8%). For all other modules did not exceed 40%.
- For all records number (100, 1k,10k, 25k, 50k), 5 concurrent jobs - RDS CPU utilization did not exceed 32%.
Results
Test Runs
Items App - updating item status
Record identifier files location - http://carrier-io.int.folio.ebsco.com/artifacts?q=export
| 5 concurrent jobs | Nolana | Morning Glory | ||||||
"BARCODE". Records number per 1 user | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes | Total time | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes | Total time |
| 100 | 8 sec | 2 sec | 10 sec | 18 sec | 10-12 sec | 2 sec | 12-13 sec | 25-27 sec |
| 1000 | 1 min 10 sec | 24-30 sec | 1 min 25 sec | 3 min | 1min 40 sec | 2-10 sec | 2 min 15 sec | 4 min |
| 10k | 10 min 45 sec | 6-7 sec | 16-17min | 28 min | 11 min 05 sec | 2-3 sec | 17-21 min | 30 min |
| 25k | 28 min | 20 sec | 35 min | 1 hour 3 min | 21 min | 6 sec | 29-30 min | 50 min |
| 50k | 3 of 5 successful - 56 min (files have 50k matched records) 2 of 5 failed "Connection reset (SocketException)" | 41 sec | 1 job- 50k records were processed in 44 min ({ For 2 of 3 jobs only 25k records were updated (UPDATED files have 25k records { | about 2 hours for successful jobs. | "Connection reset (SocketException)" | - | - | - |
* "-" test was not performed due to errors that occurred
Memory usage
The memory of mod-data-export-worker increased from 88% at the start of the tests and was about 102% at the end. Each step of memory increase is related to CPU spikes at the beginning of the records update. mod-inventory-storage memory increased from 80% to 94% (the possible reason is stepped increasing while another process starts on the server (indexing and EDIFACT)). mod-inventory memory increased by 2% (from 60% to 62% and become stable) after 2 of 3 50k records updating jobs were finished and only 1 job remaining. All other modules' memory usage was stable.
CPU utilization
CPU - for 5 parallel jobs - for mod-data-export-worker are spiking up to 120% at the start of records updating general usage was about 8%. For all other modules did not exceed 40%.
RDS CPU utilization
For all records number (100, 1k,10k, 25k, 50k), 5 concurrent jobs - RDS CPU utilization did not exceed 32%.
Tests for 25k records 5 concurrent jobs
- 25k records per user, 5 users simultaneously (125k records total) can be uploaded in about 28 min, and edited in 35 minutes (about 1 hour total).
- CPU - for all modules did not exceed 40%.
- The memory of mod-data-export-worker increased from 77% at the start of the tests and was about 89% at the end. Each step of memory increase is related to the records update PERF-328 - Getting issue details... STATUS . All other modules' memory usage was stable.
Mod-data-export-worker -1.5.0-SNAPSHOT.76 memory usage increases - PERF-328 - Getting issue details... STATUS