Overview
This is a report for a series of Check-in-check-out test runs together with Bulk edits + Data Import in the Nolana release to see how these workflow interact with one another, especially because they all use mod-inventory.
Infrastructure
PTF -environment ncp3
- 10 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-authtoken | 2.12.0 | 1 | 2 | 512 (128 in MG) | 1440 (1024 in MG) | 1152 (896 in MG) | 128 | 922 (768 in MG) |
| mod-circulation | 23.3.0 | 1 | 2 | 1024 | 1024 | 896 | 128 | 768 |
| mod-circulation-storage | 15.0.0 | 1 | 2 | 1024 | 1536 (1152 in MG) | 1440 (1024 in MG) | 512 | 896 |
| mod-configuration | 5.9.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-data-import | 2.6.1 | 1 | 1 | 256 | 2048 | 1844 | 512m | 1292m |
| mod-data-import-cs | 1.15.1 | 1 | 2 | 258 | 1024 | 896 | 128m | 768m |
| mod-feesfines | 18.1.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-inventory | 19.0.1 | 1 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.1 | 1 | 2 | 1024 | 2208 (1872 in MG) | 1952 (1684 in MG) | 512m | 1440m |
| mod-patron-blocks | 1.7.1 | 1 | 2 | 1024 | 1024 | 896 | 128 | 768 |
| mod-pubsub | 2.7.0 | 1 | 2 | 1024 | 1536 (1440 in MG) | 1440 (1296 in MG) | 512 | 922 |
| mod-quick-marc | 2.5.0 | 1 | 1 | 128 | 2288 (2098 in MG) | 2176 (1920 in MG) | 512m | 1664m |
| mod-remote-storage | 1.7.0 | 1 | 2 | 128 | 1872 | 1684 | 512m | 1178m |
| mod-source-record-manager | 3.5.4 | 2 | 2 | 1024 | 4096 (2048 in MG) | 3688 (1844 in MG) | 512m (800M in MG) | 2048m (1024m in MG) |
| mod-source-record-storage | 5.5.2 | 1 | 2 | 1024 | 1536 (1440 in MG) | 1440 (1296 in MG) | 512m | 908m |
| mod-users | 19.0.0 | 1 | 2 | 258 | 1024 | 896 | 128m | 768m |
| okapi | 4.14.7 | 1 | 3 | 1024 | 1684 (1512 in MG) | 1440 (1360 in MG) | 512m | 922m |
| mod-data-export-worker | 2.0.3 | 6 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
| mod-data-export-spring | 1.5.0 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
| mod-notes | 4.0.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-agreements | 5.4.0 | 1 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Front End:
- Item Check-in (folio_checkin-7.2.0)
- Item Check-out (folio_checkout-8.2.0)
High-Level Summary
- In general, response times for CI/CO are consistently good, even better than the CI/CO + DI test without bulk editing. We can observe a decrease in response time spikes number which can cause better results.
- Compared to baseline DI time increases for updates (baseline 24 min 5 s compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
- No errors or issues were found during the run of 3 workflows (Check-in/Check-out 8 users + Data Import MARC BIB 50k.mrc + Bulk edit 10k Items or 10k Holdings records)
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation PERF-358 - Getting issue details... STATUS ).
- For all services CPU usage did not exceed 130% for tests 1 and 2 and did not exceed 170% for tests 3 and 4.
- RDS CPU usage was high up to 94%
Test Runs
Test | Virtual Users | Duration of CI/CO | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k Items editing | 30 mins | t3.medium | 3 |
2. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k holdings editing | 30 mins | t3.medium | 3 |
3. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Items editing | 30 mins | t3.medium | 3 |
| 4. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Holdings editing | 30 mins | t3.medium | 3 |
Results
Response Times (Average of all tests listed above, in seconds)
| Check-in-check-out | Bulk edit | Data Import | ||||
|---|---|---|---|---|---|---|
| Average (seconds) | Items | Holdings | MARC BIB | |||
| Check-in | Check-out | 10k records | 10k records | 50k Create | 50k Update | |
| Test 1 | 0.715 | 1.332 | 40 min | - | 20 min 19 sec | - |
| Test 2 | 0.756 | 1.383 | - | 20 min 30 sec | 21min 07 sec | - |
| Test 3 | 0.567 | 1.029 | 32 min | - | - | 33 min 33 sec |
| Test 4 | 0.552 | 0.987 | - | 9min 20 sec | - | 30 min 52 sec |
| Jobs Baseline time to compare (single process) | 0.456 | 0.698 | 28 min | 12 min 25 sec | 23 min 43 s | 24 min 5 s |
| CICO + DI MARC BIB Create time | 0.926 | 1.666 | - | - | 19min 28s | - |
| CICO + DI MARC BIB Update time | 0.700 | 1.199 | - | - | - | 27min 39s |
| CICO + Holdings Bulk Edit time | 0.428 | 0.790 | - | 11 min 20 sec | - | - |
| CICO + Items Bulk Edit time | 0.431 | 0.793 | 20min * | - | - | - |
* - Before the Combined Bulk edit + Check-in-check-out + Data Import test, the Items Bulk edit scenario was changed to the latest requirements so the duration for bulk edits could differ.
Response times for CI/CO are consistently good, even better than the CI/CO + DI test without bulk editing. We can observe a decrease in response time spikes number which can cause better results.
Compared to baseline DI time increases for updates (baseline 24 min 5 s compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
Memory Utilization
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation PERF-358 - Getting issue details... STATUS ).
Modules CPUs Utilization
Instance CPU Utilization
Database and network
RDS CPU usage was high up to 94%
Memory & CPU Utilization table
| Nolana | CPU Avg | Memory Max |
|---|---|---|
| mod-users | 19% | 27% |
| mod-pubsub | 4% | 32% |
| okapi | 40% | 45% |
| mod-circulation | 1% | 67% |
| mod-circulation-storage | 1.5% | 32% |
| mod-inventory | 165% | 66% |
| mod-inventory-storage | 35% | 43% |
| mod-patron-blocks | 0.5% | 34% |
| mod-feesfines | 5% | 28% |
| mod-authtoken | 10% | 16% |
| mod-data-import-cs | 130% | 27% |
| mod-quik-marc | 88% | 21% |
| nginx-okapi | 73% | 3% |
| mod-source-record-storage | 46% | 60% |
| mod-source-record-manager | 41% | 73% |
| mod-remote-storage | 29% | 29% |
| mod-configuration | 8% | 24% |
| mod-data-export-worker | 5% | 81% |
| mod-graphql | 3.5% | 46% |
| mod-data-import | 80% at the start | 23% |
| pub-okapi | 3% | 2.6% |
| mod-data-export-spring | 2% | 62% |
| mod-erm-usage-harvester | 3% | 31% |
All other modules' CPU usage was less than 2%.
Data is from test #4