Bulk Edit Items App report [Nolana] 27/10/2022
Overview
Bulk Edit - Establish a performance baseline for user status bulk updates.
How long does it take to export 100, 1000, 2500, 10k, and 100K records?
Use it for up to 5 concurrent users.
Look for a memory trend and CPU usage
Infrastructure
PTF -environment
9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfka-1
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=truec
log.retention.minutes=480
default.replication.factor=3
Modules memory and CPU parameters
# | modules | task definition | running tasks | CPU | memory | memoryReservation | maxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-data-export-spring | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
| mod-data-export-worker | 3 | 2 | 1024 | 3072 | 2600 | 612 | 2128 (1536 in MG) |
| mod-users | 4 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 |
| mod-notes | 2 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) |
| mod-inventory | 2 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
| mod-inventory-storage | 5 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
| okapi | 2 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
| mod-orders | 3 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
| mod-orders-storage | 2 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-circulation | 3 | 2 | 1024 | 1024 | 896 | 128m | 768m |
| mod-circulation-storage | 6 | 2 | 1024 | 1536 | 1440 | 512m | 896m |
| mod-agreements | 2 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Software Versions
mod-data-export-worker-1.5.0-SNAPSHOT.76
mod-data-export-spring-1.5.0-SNAPSHOT.58
mod-agreements-5.4.0-SNAPSHOT.104
mod-notes-4.0.0-SNAPSHOT.237
mod-users-19.0.0-SNAPSHOT.573
mod-inventory-19.0.0-SNAPSHOT.383
mod-inventory-storage-25.0.0-SNAPSHOT.631
okapi-4.14.4
Summary
Test report for Bulk Edits items-app functionality 2022-10-27.
10k records per user, 5 users simultaneously (50k records total) can be uploaded in about 10 min 45 seconds, and edited in 17 minutes (about 28 min total).
The files with identifiers should be strictly determined.
The memory of mod-data-export-worker increased from 88% at the start of the tests and was about 102% at the end. Each step of memory increase is related to CPU spikes at the beginning of the records update. mod-inventory-storage memory increased from 80% to 94% (the possible reason is stepped increasing while another process starts on the server (indexing and EDIFACT)). mod-inventory memory increased by 2% (from 60% to 62% and become stable) after 2 of 3 50k records updating jobs were finished and only 1 job remaining. All other modules' memory usage was stable.
CPU - for 5 parallel jobs - for mod-data-export-worker are spiking up to 120% at the start of records updating( in general usage was about 8%). For all other modules did not exceed 40%.
For all records number (100, 1k,10k, 25k, 50k), 5 concurrent jobs - RDS CPU utilization did not exceed 32%.
Results
Test Runs
Items App - updating item status
Record identifier files location - http://carrier-io.int.folio.ebsco.com/artifacts?q=export
5 concurrent jobs | Nolana | Morning Glory | ||||||
"BARCODE". Records number per 1 user | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes | Total time | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes | Total time |
100 | 8 sec | 2 sec | 10 sec | 18 sec | 10-12 sec | 2 sec | 12-13 sec | 25-27 sec |
1000 | 1 min 10 sec | 24-30 sec | 1 min 25 sec | 3 min | 1min 40 sec | 2-10 sec | 2 min 15 sec | 4 min |
10k | 10 min 45 sec | 6-7 sec | 16-17min | 28 min | 11 min 05 sec | 2-3 sec | 17-21 min | 30 min |
25k | 28 min | 20 sec | 35 min | 1 hour 3 min | 21 min | 6 sec | 29-30 min | 50 min |
50k | 3 of 5 successful - 56 min (files have 50k matched records) 2 of 5 failed "Connection reset (SocketException)" | 41 sec | 1 job- 50k records were processed in 44 min ({ For 2 of 3 jobs only 25k records were updated (UPDATED files have 25k records { | about 2 hours for successful jobs. | "Connection reset (SocketException)" | - | - | - |
* "-" test was not performed due to errors that occurred
Memory usage
The memory of mod-data-export-worker increased from 88% at the start of the tests and was about 102% at the end. Each step of memory increase is related to CPU spikes at the beginning of the records update. mod-inventory-storage memory increased from 80% to 94% (the possible reason is stepped increasing while another process starts on the server (indexing and EDIFACT)). mod-inventory memory increased by 2% (from 60% to 62% and become stable) after 2 of 3 50k records updating jobs were finished and only 1 job remaining. All other modules' memory usage was stable.
CPU utilization
CPU - for 5 parallel jobs - for mod-data-export-worker are spiking up to 120% at the start of records updating general usage was about 8%. For all other modules did not exceed 40%.
RDS CPU utilization
For all records number (100, 1k,10k, 25k, 50k), 5 concurrent jobs - RDS CPU utilization did not exceed 32%.
Tests for 25k records 5 concurrent jobs
25k records per user, 5 users simultaneously (125k records total) can be uploaded in about 28 min, and edited in 35 minutes (about 1 hour total).
CPU - for all modules did not exceed 40%.
The memory of mod-data-export-worker increased from 77% at the start of the tests and was about 89% at the end. Each step of memory increase is related to the records update
PERF-328: Investigate memory usage for mod-data-export-workerClosed. All other modules' memory usage was stable.
Mod-data-export-worker -1.5.0-SNAPSHOT.76 memory usage increases -