IN PROGRESS
Overview
Bulk Edit - Establish a performance baseline for user status bulk updates.
- How long does it take to export 100, 1000, 2500, 10k, and 100K records?
- Use it for up to 5 concurrent users.
- Look for a memory trend and CPU usage
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfka-1
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
| # | modules | task definition | running tasks | CPU | memory | memoryReservation | maxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-data-export-spring | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) | |
| mod-data-export-worker | 2 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) | |
| mod-users | 4 | 2 | 128 (258 in MG) | 1024 | 896 | 128 | 768 | |
| mod-notes | 2 | 2 | 128 | 1024 | 896 | 128m | 322 (768m in MG) | |
| mod-inventory | 2 | 2 | 1024 | 2880 | 2592 | 512m | 1814m | |
mod-inventory-storage | 5 | 2 | 1024 | 2208 | 1952 | 512m | 1440m | |
| okapi | 2 | 3 | 1024 | 1684 | 1440 | 512m | 922m | |
| mod-orders | 3 | 2 | 1024 | 2048 | 1440 | 512m | 896m | |
| mod-orders-storage | 2 | 2 | 128 | 1024 | 896 | 128m | 768m | |
| mod-circulation | 3 | 2 | 1024 | 1024 | 896 | 128m | 768m | |
| mod-circulation-storage | 6 | 2 | 1024 | 1536 | 1440 | 512m | 896m | |
| mod-agreements | 2 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Software Versions
- mod-data-export-worker-1.5.0-SNAPSHOT.76
- mod-data-export-spring-1.5.0-SNAPSHOT.58
- mod-agreements-5.4.0-SNAPSHOT.104
- mod-notes-4.0.0-SNAPSHOT.237
- mod-users-19.0.0-SNAPSHOT.573
- mod-inventory-19.0.0-SNAPSHOT.383
- mod-inventory-storage-25.0.0-SNAPSHOT.631
- okapi-4.14.4
Summary
Test report for Bulk Edits users-app functionality 2022-10-29.
- 10k records per user, 5 users simultaneously (50k records total) can be uploaded in about 3 min 15 seconds, edited files uploaded in up to 15 seconds, and edited in 6 min 30 seconds (about 10 min total).
- The files with identifiers should be strictly determined.
- The memory of all modules during the tests for 2500 records with 5 parallel was stable, and during the tests for 5000 (5 parallel) and 10 000 records (5 parallel) jobs by 2% (from 43% to 45%). For mod-data-export-worker memory was about 97% all the time. All other modules' memory usage was stable.
- CPU - 2500 records per user for 5 parallel jobs -CPU of mod-users was 199%, for all other modules did not exceed 15%.
- RDS CPU utilization did not exceed 36% for 5jobs 2500 records
Results
Test Runs
Users App - updating user status
Record identifier files location - http://carrier-io.int.folio.ebsco.com/artifacts?q=export
| 1 virtual user | "BARCODE" | ||
Records number per 1 user | Time to upload | Time of POST /bulk-edit/${jobId}/items-content-update/upload | Time to edit |
| 100 | 2 sec | 2 sec | 2 sec |
| 1000 | 12 sec | 8 sec | 18 sec |
| 2500 | 31 sec | 4.7 min MODEXPW-255 - Getting issue details... STATUS | 21 sec |
| 10k | 40 min if 0 records matched related to MODEXPW-255 - Getting issue details... STATUS | - | - |
* "-'' test was not performed due to 0 records can be updated
| 5 virtual users | Nolana | Morning Glory | ||||
"BARCODE" Records number per 1 user | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes | Time to upload | Time of POST /bulk-edit/${jobId}/upload | Time to edit - commit changes |
| 2500 | 40sec | 6 sec | 1 min 23 sec | 34-39 sec | up to 10 minutes (depending on the number of records that should not be updated) | 1 min 5 sec |
| 5000 | 1min 12 sec | 10 sec | 2 min 25 sec | 1 min 12 sec | 35 sec if all records will be updated MODEXPW-255 - Getting issue details... STATUS | 1min 50 sec |
| 10k | 2 min 25 sec | up to 15 sec | 4 min 50 sec | 2min 15 sec | 2 min if all records will be updated | 3 min 58 sec |
Memory usage
The memory of mod-users increases during the tests for 2500 records 5 parallel jobs by 1%(from 42% to 43%) and during the tests for 5000 (5 parallel) and 10 000 records (5 parallel) jobs by 2%(from 43% to 45%). For mod-data-export-worker memory was about 97% all the time.
CPU utilization
2500 records per user 5 parallel jobs:
| modules | max CPU utilization |
|---|---|
| mod-users | 116% |
| mod-data-export-worker | 19% |
| nginx-okapi | 13% |
| okapi | 11% |
| mod-data-export-spring | 7% |
| mod-inventory | 6% |
| mod-agreements | 4% |
| mod-notes | 3% |
| mod-inventory-storage | 0.5% |
5000 & 10k records 5 parallel jobs -CPU of modules did not exceed 149%
RDS CPU utilization
2500 records - RDS CPU utilization did not exceed 42%
5000 & 10k records - RDS CPU utilization did not exceed 52%
