Overview
This is a report for a series of Check-in-check-out test runs together with Bulk edits + Data Import in the Nolana release to see how these workflow interact with one another, especially because they all use mod-inventory.
Infrastructure
PTF -environment ncp3
- 10 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-authtoken | 2.12.0 | 1 | 2 | 512 (128 in MG) | 1440 (1024 in MG) | 1152 (896 in MG) | 128 | 922 (768 in MG) |
| mod-circulation | 23.3.0 | 1 | 2 | 1024 | 1024 | 896 | 128 | 768 |
| mod-circulation-storage | 15.0.0 | 1 | 2 | 1024 | 1536 (1152 in MG) | 1440 (1024 in MG) | 512 | 896 |
| mod-configuration | 5.9.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-data-import | 2.6.1 | 1 | 1 | 256 | 2048 | 1844 | 512m | 1292m |
| mod-data-import-cs | 1.15.1 | 1 | 2 | 258 | 1024 | 896 | 128m | 768m |
| mod-feesfines | 18.1.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-inventory | 19.0.1 | 1 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
mod-inventory-storage | 25.0.1 | 1 | 2 | 1024 | 2208 (1872 in MG) | 1952 (1684 in MG) | 512m | 1440m |
| mod-patron-blocks | 1.7.1 | 1 | 2 | 1024 | 1024 | 896 | 128 | 768 |
| mod-pubsub | 2.7.0 | 1 | 2 | 1024 | 1536 (1440 in MG) | 1440 (1296 in MG) | 512 | 922 |
| mod-quick-marc | 2.5.0 | 1 | 1 | 128 | 2288 (2098 in MG) | 2176 (1920 in MG) | 512m | 1664m |
| mod-remote-storage | 1.7.0 | 1 | 2 | 128 | 1872 | 1684 | 512m | 1178m |
| mod-source-record-manager | 3.5.4 | 2 | 2 | 1024 | 4096 (2048 in MG) | 3688 (1844 in MG) | 512m (800M in MG) | 2048m (1024m in MG) |
| mod-source-record-storage | 5.5.2 | 1 | 2 | 1024 | 1536 (1440 in MG) | 1440 (1296 in MG) | 512m | 908m |
| mod-users | 19.0.0 | 1 | 2 | 258 | 1024 | 896 | 128m | 768m |
| okapi | 4.14.7 | 1 | 3 | 1024 | 1684 (1512 in MG) | 1440 (1360 in MG) | 512m | 922m |
| mod-data-export-worker | 2.0.3 | 6 | 2 | 1024 | 3072 | 2600 | 512 | 2048 (1536 in MG) |
| mod-data-export-spring | 1.5.0 | 2 | 1 | 256 | 2048 | 1844 | 512 | 1536 (1236 in MG) |
| mod-notes | 4.0.0 | 1 | 2 | 128 | 1024 | 896 | 128m | 768m |
| mod-agreements | 5.4.0 | 1 | 2 | 128 | 1592 | 1488 | 512m | 968m |
MG- Morning Glory release
Front End:
- Item Check-in (folio_checkin-7.2.0)
- Item Check-out (folio_checkout-8.2.0)
High-Level Summary
- In general, response times for CI/CO are consistently good, even better than the CI/CO + DI test without bulk editing. We can observe a decrease in response time spikes number which can cause better results.
- Compared to baseline DI time increases for updates (baseline 24 min 5 s compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
- No errors or issues were found during the run of 3 workflows (Check-in/Check-out 8 users + Data Import MARC BIB 50k.mrc + Bulk edit 10k Items or 10k Holdings records)
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation Error rendering macro 'jira' : Unable to locate Jira server for this macro. It may be due to Application Link configuration.).
- For all services CPU usage did not exceed 130% for tests 1 and 2 and did not exceed 170% for tests 3 and 4.
- RDS CPU usage was high up to 94%
Test Runs
Test | Virtual Users | Duration of CI/CO | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k Items editing | 30 mins | t3.medium | 3 |
2. | 8 users CI/CO + DI 50k MARC BIB Create+ 10k holdings editing | 30 mins | t3.medium | 3 |
3. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Items editing | 30 mins | t3.medium | 3 |
| 4. | 8 users CI/CO + DI 50k MARC BIB Update+ 10k Holdings editing | 30 mins | t3.medium | 3 |
Results
Response Times (Average of all tests listed above, in seconds)
| Check-in-check-out | Bulk edit | Data Import | ||||
|---|---|---|---|---|---|---|
| Average (seconds) | Items | Holdings | MARC BIB | |||
| Check-in | Check-out | 10k records | 10k records | 50k Create | 50k Update | |
| Test 1 | 0.715 | 1.332 | 40 min | - | 20 min 19 sec | - |
| Test 2 | 0.756 | 1.383 | - | 20 min 30 sec | 21min 07 sec | - |
| Test 3 | 0.567 | 1.029 | 32 min | - | - | 33 min 33 sec |
| Test 4 | 0.552 | 0.987 | - | 9min 20 sec | - | 30 min 52 sec |
| Jobs Baseline time to compare (single process) | 0.456 | 0.698 | 28 min | 12 min 25 sec | 23 min 43 s | 24 min 5 s |
| CICO + DI MARC BIB Create time | 0.926 | 1.666 | - | - | 19min 28s | - |
| CICO + DI MARC BIB Update time | 0.700 | 1.199 | - | - | - | 27min 39s |
| CICO + Holdings Bulk Edit time | 0.428 | 0.790 | - | 11 min 20 sec | - | - |
| CICO + Items Bulk Edit time | 0.431 | 0.793 | 20min * | - | - | - |
* - Before the Combined Bulk edit + Check-in-check-out + Data Import test, the Items Bulk edit scenario was changed to the latest requirements so the duration for bulk edits could differ.
Response times for CI/CO are consistently good, even better than the CI/CO + DI test without bulk editing. We can observe a decrease in response time spikes number which can cause better results.
Compared to baseline DI time increases for updates (baseline 24 min 5 s compared to test #4 and Test #3 about 30 min). Bulk edit time increases for tests with DI Create (about 10 min slower).
Memory Utilization
- Memory usage is stable for all modules, except for mod-source-record-manager, mod-source-record-storage, and mod-inventory increased (reason: modules were restarted manually before the tests. No memory leaks were found during modules memory leaks investigation ).
Modules CPUs Utilization
Instance CPU Utilization
Database and network
RDS CPU usage was high up to 94%
Memory & CPU Utilization table
| Nolana | CPU Avg | Memory Max |
|---|---|---|
| mod-users | 19% | 27% |
| mod-pubsub | 4% | 32% |
| okapi | 40% | 45% |
| mod-circulation | 1% | 67% |
| mod-circulation-storage | 1.5% | 32% |
| mod-inventory | 165% | 66% |
| mod-inventory-storage | 35% | 43% |
| mod-patron-blocks | 0.5% | 34% |
| mod-feesfines | 5% | 28% |
| mod-authtoken | 10% | 16% |
| mod-data-import-cs | 130% | 27% |
| mod-quik-marc | 88% | 21% |
| nginx-okapi | 73% | 3% |
| mod-source-record-storage | 46% | 60% |
| mod-source-record-manager | 41% | 73% |
| mod-remote-storage | 29% | 29% |
| mod-configuration | 8% | 24% |
| mod-data-export-worker | 5% | 81% |
| mod-graphql | 3.5% | 46% |
| mod-data-import | 80% at the start | 23% |
| pub-okapi | 3% | 2.6% |
| mod-data-export-spring | 2% | 62% |
| mod-erm-usage-harvester | 3% | 31% |
All other modules' CPU usage was less than 2%.
Data is from test #4