Data Import Test Report OL enabled + background activities
Overview
This document contains the results of testing Data Import in Kiwi with OL enabled + background activities (Check-in/Check-out test with 8 users).
Infrastructure
- 6 m5.xlarge EC2 instances
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- mod-inventory memory
- 256 CPU units, 1814MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 128 CPU units, 544MB mem
- mod-source-record-storage
- 128 CPU units, 908MB mem
- mod-source-record-manager
- 128 CPU units, 1292MB mem
- mod-data-import
- 128 CPU units, 1024MB mem
Software versions
- mod-data-import:2.2.0
- mod-inventory:18.0.4
- mod-inventory-storage:22.0.2-optimistic-locking.559
- mod-source-record-storage:5.2.5
- mod-source-record-manager:3.2.6
Results
Tests performed:
| Profile | Duration KIWI WO background activities | Duration | Began UTC | CheckIn average | CheckOut average | Link to CheckIn-CheckOut chart | |
|---|---|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 5 min, 8 min (05:32.264) | 5 min | 10:42 AM 022.01.10 | 1.297 | 2.449 | click to jump |
5K MARC Update | PTF - Updates Success - 1 | 11 min, 13 min | 7 min | 11:20 AM 2022.01.10 | 1.315 | 2.469 | click to jump |
10K MARC Create | PTF - Create 2 | 11 min , 14 min | 13 min | 2:44 PM 2022.01.10 | 1.407 | 2.795 | click to jump |
| 10K MARC Update | PTF - Updates Success - 1 | 22 min, 24 min | 15 min | 3:26 PM 2022.01.10 | 1.691 | 3.041 | click to jump |
| 25K MARC Create | PTF - Create 2 | 23 mins, 25 mins, 26 mins | 31 min | 8:55 AM 2022.01.11 | 1.536 | 2.962 | click to jump |
| 25K MARC Update | PTF - Updates Success - 1 | 27 min, 40 mins, 56mins | 33 min | 9:56 AM 2022.01.11 | 1.589 | 2.997 | click to jump |
| 50K MARC Create | PTF - Create 2 | 40 mins, 43mins, | 63 min | 12:45 PM 2022.01.12 | 1.783 | 3.231 | click to jump |
| 50K MARC Update | PTF - Updates Success - 1 | 64 min | FAILED | 3:56 PM 2022.01.12 | - | - |
Вelow are the general timings for check-in check-out. The horizontal axis shows the number of mrs records in thousands
We can see her an exponential increase of the import time depending on the file size
also there a slight linear increase in the average latency for the check-in check-out
It can be seen from the graphs below that - after the start of the data-import, the check-in response time increases sharply from about 0.6 seconds to 3 seconds,
the check-out time increases accordingly from 1 to 6 seconds, then further the time gradually returns to its previous value during the entire test process.
check-in check out time chart for 5K create
at the start of the date-import 5K, delays for the CI increase from average 0.8 up to 3 sec sharply , then gradually return to its average value 0.8 sec during 5 min (all time of data import)
for CI same process with same times but values are greater of delay in 1.8 up to 5 sec than return to average during 5 min
| 5K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 3 sec | 5 min |
| Check out | 1.8 sec | 5 sec | 5 min |
check-in check out time chart for 5K update
check-in check out time chart for 10K create
| 10K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 5 sec | 11 min |
| Check out | 1.8 sec | 7 sec | 11 min |
check-in check out time chart for 10K update
check-in check out time chart for 25K create
| 25K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 3 sec | 15 min |
| Check out | 1.8 sec | 5 sec | 15 min |
heck-in check out time chart for 25K update
check-in check out time chart for 50K create
| 50K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 5 sec | 15 min |
| Check out | 1.8 sec | 8 sec | 15 min |
CPU utilisation graph: 25K create - 25K update
Instance CPU level 25K create-update
We see a step increase in load during the test for each instance, but no more than 80%
next chart - instance CPU levels 25K create without background activities for comparison
we see on the next fragments a significant increase in utilization (up to 90%) on two instances in the case of background activity
DB CPU level for 25K create-update
Service CPU utilization 25K create-update
Graphs for different services are separated because they have different scales. But in general, everything looks like there is no CPU overload on any of the services. the graph shows increase CPU load after the start of the service by modules
Service Memory usage 50K create
Instance CPU levels 50k create
In the 50k test we see a flat stepped load across instances, in shape corresponding to the dynamic load distribution of java VM
Service CPU levels 50k create
DB CPU Level 50Kcreate
I was not able to find out where the peaks on the reader came from, the repeated test did not show spikes not at 25K creation, not at 50K
On this graph we can see CPU DB without backgrount activity
And this in is graph of DB CPU with check-in check-out. Here we can see large fluctuations in the CPU load
Kafka topics messages' rates during 50K create
Because the SRS records are created first before instances, holdings, and items are created. That's why mod-srs CPU utilization spikes in the first minutes of the import.
The spike for kcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_CREATED immediately after the start, then the activity on this topic completely stops .
