...
This is a report for a series of Check-in-check-out test runs together with Bulk edits + Data Import in the Nolana release to see how these workflow workflows interact with one another, especially because they all use mod-inventory.
Infrastructure
PTF -environment ncp3
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
- In general, response times for CI/CO are consistently good, even better than the CI/CO + DI test without bulk editing. We can observe a decrease in response time spikes number which can cause better results.
- Compared to baseline DI time increases for updates (baseline 24 20 min 5 57 s compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
- No errors or issues were found during the run of 3 workflows (Check-in/Check-out 8 users + Data Import MARC BIB 50k.mrc + Bulk edit 10k Items or 10k Holdings records)
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation
).Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-358 - For all services CPU usage did not exceed 130% for tests 1 and 2 and did not exceed 170% for tests 3 and 4.
- RDS CPU usage was high up to 94%
...
Test | Virtual Users | Duration of CI/CO | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k Items editing | 30 mins | t3.medium | 3 |
2. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k holdings editing | 30 mins | t3.medium | 3 |
3. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Items editing | 30 mins | t3.medium | 3 |
| 4. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Holdings editing | 30 mins | t3.medium | 3 |
Results
Response Times (Average of all tests listed above, in seconds)
| Check-in-check-out | Bulk edit | Data Import | ||||
|---|---|---|---|---|---|---|
| Average (seconds) | Items | Holdings | MARC BIB | |||
| Check-in | Check-out | 10k records | 10k records | 50k Create | 50k Update | |
| Test 1 | 0.715 | 1.332 | 40 min | - | 20 min 19 sec | - |
| Test 2 | 0.756 | 1.383 | - | 20 min 30 sec | 21min 07 sec | - |
| Test 3 | 0.567 | 1.029 | 32 min | - | - | 33 min 33 sec |
| Test 4 | 0.552 | 0.987 | - | 9min 20 sec | - | 30 min 52 sec |
| Jobs Baseline time to compare (single process) | 0.456 | 0.698 | 28 min | 12 min 25 sec23 min 43 s | 21 min 11 s (for mod-source-record-manager v. 3.5.4) | 20 min 57 s (for mod-source-record-manager v. 3.5.4) |
| CICO + DI MARC BIB Create time | 0.926 | 1.666 | - | - | 19min 28s | - |
| CICO + DI MARC BIB Update time | 0.700 | 1.199 | - | - | - | 27min 39s |
| CICO + Holdings Bulk Edit time | 0.428 | 0.790 | - | 11 min 20 sec | - | - |
| CICO + Items Bulk Edit time | 0.431 | 0.793 | 20min * | - | - | - |
...
Compared to baseline DI time increases for updates (baseline 24 20 min 5 57 s compared compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
...
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation
).Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-358
...