Data Import (MARC Holdings) with CI/CO [Nolana]
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.
Overview
The target of the test is to investigate the impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations and to compare tested results with direct non-intersecting activities' baselines.
- PERF-291Getting issue details... STATUS
Summary
- Test results don't show a huge negative impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations:
- Response time of CI/CO activities during Data Import - MARC Holdings was increased (degradation) by more than 25%.
- Duration of Data Import - MARC Holdings with CI/CO activities was decreased by 66%.
- High Memory usage was detected for mod-inventory-storage (memory usage is 106%, it's >100%) but stable during the test
- Memory usage of mod-source-record-storage was increased from 55% to 58%
- All modules show expected CPU behaviour and typical workload, except 'mod-source-record-manager' and 'mod-authtoken'
- During the test, there were detected several requests with more than a 1-second response time. (see Additional Screenshots of graphs or charts)
Recommendations & Jiras
- During future tests, pay attention to:
- Memory usage of mod-inventory-storage, mod-source-record-storage
- CPU usage of mod-source-record-manager, mod-authtoken
- To discuss Data Import - MARC Holdings' behaviour during Check-In/Check-Out activities Martin Tran Artem Vronik Roman_Fedynyshyn
Test Runs
Test # | Test Conditions | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 8 users CI/CO + DI (1k, 5k, 10k, 80k) MARC Holding | 60 min | t3.medium | 3 |
Results
Response Times
Check-in/Check-out | Data Import | |||||
---|---|---|---|---|---|---|
Average, ms | MARC Holding | |||||
Check-in Controller | Check-out Controller | 1k Records | 5k Records | 10k Records | 80k records | |
Test 1 | 573.79 | 924.19 | "00:00:35.777" | "00:01:28.613" | "00:02:42.095" | "00:20:12.114" |
Comparisons
Test # | Report link | Type of results | Check-In, ms | Check-Out, ms | DI MARC Holding | |||
1k | 5k | 10k | 80k | |||||
Test #1 - CI/CO 8 users only | Check-in-check-out Test Report (Nolana) | [Baseline] | 456.00 | 698.00 | x | x | x | x |
Test #2 - DI MARC Holding only | Data Import Create MARC holdings records [Nolana] | [Baseline] | x | x | 00:00:32.700 | 00:04:20.800 | 00:03:24.900 | 00:21:22.600 |
Test #3 - CI/CO + DI MARC Holding *CI/CO stable | Data Import (MARC Holdings) with CI/CO [Nolana] | [Tested] | 573.79 461.17 | 924.19 711.94 | 00:00:35.777 | 00:01:28.613 | 00:02:42.095 | 00:20:12.114 |
Diff: | +25.83% *+1.13% | +32.41% *+1.99% | +9.41% | -66.02% | -20.89% | -5.50% |
*Ci/CO stable:
Memory Utilization
mod-inventory-storage memory usage is 106%, it's >100% but stable during test
mod-source-record-storage memory usage was increased from 55% to 58%
CPU Utilization
All modules show expected and some workload, except 'mod-source-record-manager' and 'mod-authtoken':
- mod-source-record-manager: MIN - 2.74%, MAX - 50.87%, AVG - 21.674%
- mod-authtoken: MIN - 0.33%, MAX - 18.59%, AVG - 5.96%
TOP CPU usage modules:
- nginx-okapi: MIN - 0.13%, MAX - 72.06%, AVG - 33.54%
- mod-source-record-storage: MIN - 1.95%, MAX - 67.80%, AVG - 29.83%
- mod-inventory: MIN - 6.07%, MAX - 59.05%, AVG - 27.44%
RDS CPU Utilization
MAX RDS CPU utilization is 49% under 80k DI Marc holding activity.
Appendix
Infrastructure
PTF -environment ncp3 [ environment name]
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Modules | Version | Task Definition | Running Tasks | CPU | Memory (Soft/Hard Limits) | MaxMetaspaceSize | Xmx |
---|---|---|---|---|---|---|---|
mod-inventory | 19.0.2 | 7 | 2 | 1024 | 2592/2880 | 512 | 1814 |
mod-inventory-storage | 25.0.3 | 3 | 2 | 1024 | 1952/2208 | 512 | 1440 |
okapi | 4.14.7 | 1 | 3 | 1024 | 1440/1684 | 512 | 922 |
mod-source-record-storage | 5.5.2 | 4 | 2 | 1024 | 1440/1536 | 512 | 908 |
mod-source-record-manager | 3.5.6 | 4 | 2 | 1024 | 3688/4096 | 512 | 2048 |
mod-data-import | 2.6.2 | 4 | 1 | 256 | 1844/2048 | 512 | 1292 |
mod-data-import-cs | 1.15.1 | 1 | 2 | 128 | 896/1024 | 128 | 768 |
mod-quick-marc | 2.5.0 | 3 | 1 | 128 | 2176/2288 | 512 | 1664 |
mod-feesfines | 18.1.1 | 3 | 2 | 128 | 896/1024 | 128 | 768 |
mod-patron-blocks | 1.7.1 | 4 | 2 | 1024 | 896/1024 | 128 | 768 |
mod-pubsub | 2.7.0 | 4 | 2 | 1024 | 1440/1536 | 512 | 922 |
mod-authtoken | 2.12.0 | 3 | 2 | 512 | 1152/1440 | 128 | 922 |
mod-circulation-storage | 15.0.2 | 3 | 2 | 1024 | 1440/1536 | 512 | 896 |