Overview
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously.
Summary
Test report for Bulk Edits holdings-app functionality 2023-03-22.
Orchid release works
It is approximately the same stable as Nolana.
- For 1 concurrent job, 100 records can be edited in 1 min 9 s which is 19 s slower compared to Nolana (50 s), and 1000 records editing could be performed in 2 min 54 s which is 40 s slower compared to Nolana(2 min 10 s), and 10k records bulk editing is about 36% slower.
- 10k records per user, 5 users simultaneously (50k records total) can be uploaded and edited in about 22 minutes which is about 9 min 30 s slower compared to Nolana (about 12 min 25 s).
- The memory utilization of mod-bulk operation increases from 20% to 23% (The service was updated before the test, probably it is reaching a steady state- the memory trend will be investigated in further testing). For all other modules, no memory leaks are suspected.
- CPU for all modules did not exceed 56% for all of the tests.
- For all records number (100, 1k,10k), and 5 concurrent jobs - RDS CPU utilization did not exceed 41%.
Recommendations & Jiras
Results
For all of the tests used the BARECODE identifier type for ITEMS & USERS and the HRID identifier type for Holdings.
Test Runs
1# 2 concurrent jobs with 10k holdings records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k holding records | 18 min 35 s | 19 min 33 s |
| 2 | 2500 user records | 3 min 37 s | 3 min 32 s |
2# 2 concurrent jobs with 10k item records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 41 s | 17 min 50 s |
| 2 | 2500 user records | 3 min 38 s | 3 min 32 s |
3# 3 concurrent jobs with 10k item records + 10k holdings records + 2500 user records
| 3 concurrent jobs | |||
| Job # | Job description | Job duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 43 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 33 s | 19 min 33 s |
| 3 | 2500 user records | 3 min 34 s | 3 min 32 s |
4# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 18 min 19 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 35 s | 19 min 33 s |
5# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 49 s | 17 min 50 s |
| 2 | 10k user records | 11 min 26 s | 11 min 17 s |
6# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k holding records | 18 min 34 s | 19 min 33 s |
| 2 | 10k user records | 13 min 9 s | 11 min 17 s |
7# 3 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 48 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 56 s | 19 min 33 s |
| 3 | 10k user records | 11 min 45 s | 11 min 17 s |
Memory usage
Instance CPU utilization
Service CPU utilization
RDS CPU utilization
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Module | SoftLimit | XMX | Revision | Version | desiredCount | CPUUnits | RWSplitEnabled | HardLimit | Metaspace | MaxMetaspaceSize |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage-b | 1952 | 1440 | 3 | mod-inventory-storage:26.1.0-SNAPSHOT.644 | 2 | 1024 | False | 2208 | 384 | 512 |
| mod-inventory-b | 2592 | 1814 | 7 | mod-inventory:20.0.0-SNAPSHOT.392 | 2 | 1024 | False | 2880 | 384 | 512 |
| okapi-b | 1440 | 922 | 1 | okapi:5.1.0-SNAPSHOT.1352 | 3 | 1024 | False | 1684 | 384 | 512 |
| mod-users-b | 896 | 768 | 4 | mod-users:19.2.0-SNAPSHOT.584 | 2 | 128 | False | 1024 | 88 | 128 |
| mod-data-export-worker | 2600 | 2048 | 3 | mod-data-export-worker:3.0.0-SNAPSHOT.104 | 2 | 1024 | False | 3072 | 384 | 512 |
| mod-data-export-spring | 1844 | 1292 | 3 | mod-data-export-spring:2.0.0-SNAPSHOT.67 | 1 | 256 | False | 2048 | 200 | 256 |
| mod-bulk-operations | 3864 | 0 | 10 | mod-bulk-operations:1.0.2 | 2 | 400 | False | 4096 | 384 | 512 |
| mod-notes | 896 | 322 | 3 | mod-notes:5.1.0-SNAPSHOT.245 | 2 | 128 | False | 1024 | 128 | 128 |
| mod-agreements | 2580 | 2048 | 3 | mod-agreements:5.6.0-SNAPSHOT.117 | 2 | 128 | False | 3096 | 384 | 512 |
| nginx-okapi | 896 | 0 | 3 | nginx-okapi:2022.03.02 | 2 | 128 | False | 1024 | 0 | 0 |