Overview
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously.
Summary
Test report for Bulk Edits holdings-app functionality 2023-03-22.
Orchid release works
It is approximately the same stable as Nolana.
- The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing). mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests. For all other modules, no memory leaks are suspected.
- CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
- For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Recommendations & Jiras
Results
For all of the tests used the BARECODE identifier type for ITEMS & USERS and the HRID identifier type for Holdings.
Test Runs
1# 2 concurrent jobs with 10k holdings records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k holding records | 18 min 35 s | 19 min 33 s |
| 2 | 2500 user records | 3 min 37 s | 3 min 32 s |
2# 2 concurrent jobs with 10k item records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 41 s | 17 min 50 s |
| 2 | 2500 user records | 3 min 38 s | 3 min 32 s |
3# 3 concurrent jobs with 10k item records + 10k holdings records + 2500 user records
| 3 concurrent jobs | |||
| Job # | Job description | Job duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 43 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 33 s | 19 min 33 s |
| 3 | 2500 user records | 3 min 34 s | 3 min 32 s |
4# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 18 min 19 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 35 s | 19 min 33 s |
5# 2 Concurrent jobs with 10k item records + 10k user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 49 s | 17 min 50 s |
| 2 | 10k user records | 11 min 26 s | 11 min 17 s |
6# 2 Concurrent jobs with 10k user records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k holding records | 18 min 34 s | 19 min 33 s |
2 | 10k user records | 13 min 9 s | 11 min 17 s |
7# 3 Concurrent jobs with 10k item records + 10k holdings records +10k user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 48 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 56 s | 19 min 33 s |
| 3 | 10k user records | 11 min 45 s | 11 min 17 s |
Memory usage
The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing).
mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests.
For all other modules, no memory leaks are suspected.
Instance CPU utilization
Instances CPU utilization did not exceed 16%
Service CPU utilization
CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
RDS CPU utilization
For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Module | SoftLimit | XMX | Revision | Version | desiredCount | CPUUnits | RWSplitEnabled | HardLimit | Metaspace | MaxMetaspaceSize |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage-b | 1952 | 1440 | 3 | mod-inventory-storage:26.1.0-SNAPSHOT.644 | 2 | 1024 | False | 2208 | 384 | 512 |
| mod-inventory-b | 2592 | 1814 | 7 | mod-inventory:20.0.0-SNAPSHOT.392 | 2 | 1024 | False | 2880 | 384 | 512 |
| okapi-b | 1440 | 922 | 1 | okapi:5.1.0-SNAPSHOT.1352 | 3 | 1024 | False | 1684 | 384 | 512 |
| mod-users-b | 896 | 768 | 4 | mod-users:19.2.0-SNAPSHOT.584 | 2 | 128 | False | 1024 | 88 | 128 |
| mod-data-export-worker | 2600 | 2048 | 3 | mod-data-export-worker:3.0.0-SNAPSHOT.104 | 2 | 1024 | False | 3072 | 384 | 512 |
| mod-data-export-spring | 1844 | 1292 | 3 | mod-data-export-spring:2.0.0-SNAPSHOT.67 | 1 | 256 | False | 2048 | 200 | 256 |
| mod-bulk-operations | 3864 | 0 | 10 | mod-bulk-operations:1.0.2 | 2 | 400 | False | 4096 | 384 | 512 |
| mod-notes | 896 | 322 | 3 | mod-notes:5.1.0-SNAPSHOT.245 | 2 | 128 | False | 1024 | 128 | 128 |
| mod-agreements | 2580 | 2048 | 3 | mod-agreements:5.6.0-SNAPSHOT.117 | 2 | 128 | False | 3096 | 384 | 512 |
| nginx-okapi | 896 | 0 | 3 | nginx-okapi:2022.03.02 | 2 | 128 | False | 1024 | 0 | 0 |