Data Import Test Report OL enabled + background activities
- Roman_Fedynyshyn
- Iakov Zaborov
Overview
This document contains the results of testing Data Import in Kiwi with OL enabled + background activities (Check-in/Check-out test with 8 users).
Infrastructure
- 6 m5.xlarge EC2 instances
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- mod-inventory memory
- 256 CPU units, 1814MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 128 CPU units, 544MB mem
- mod-source-record-storage
- 128 CPU units, 908MB mem
- mod-source-record-manager
- 128 CPU units, 1292MB mem
- mod-data-import
- 128 CPU units, 1024MB mem
Software versions
- mod-data-import:2.2.0
- mod-inventory:18.0.4
- mod-inventory-storage:22.0.2-optimistic-locking.559
- mod-source-record-storage:5.2.5
- mod-source-record-manager:3.2.6
Results
Tests performed:
| Profile | Duration KIWI WO background activities | Duration | Began UTC | CheckIn average | CheckOut average | Link to CheckIn-CheckOut chart | |
|---|---|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 5 min, 8 min (05:32.264) | 5 min | 10:42 AM 022.01.10 | 1.297 | 2.449 | click to jump |
5K MARC Update | PTF - Updates Success - 1 | 11 min, 13 min | 7 min | 11:20 AM 2022.01.10 | 1.315 | 2.469 | click to jump |
10K MARC Create | PTF - Create 2 | 11 min , 14 min | 13 min | 2:44 PM 2022.01.10 | 1.407 | 2.795 | click to jump |
| 10K MARC Update | PTF - Updates Success - 1 | 22 min, 24 min | 15 min | 3:26 PM 2022.01.10 | 1.691 | 3.041 | click to jump |
| 25K MARC Create | PTF - Create 2 | 23 mins, 25 mins, 26 mins | 31 min | 8:55 AM 2022.01.11 | 1.536 | 2.962 | click to jump |
| 25K MARC Update | PTF - Updates Success - 1 | 27 min, 40 mins, 56mins | 33 min | 9:56 AM 2022.01.11 | 1.589 | 2.997 | click to jump |
| 50K MARC Create | PTF - Create 2 | 40 mins, 43mins, | 63 min | 12:45 PM 2022.01.12 | 1.783 | 3.231 | click to jump |
| 50K MARC Update | PTF - Updates Success - 1 | 64 min | FAILED | 3:56 PM 2022.01.12 | - | - |
Вelow are the general timings for check-in check-out. The horizontal axis shows the number of mrs records in thousands
We can see her an exponential increase of the import time depending on the file size
also there a slight linear increase in the average latency for the check-in check-out
It can be seen from the graphs below that - after the start of the data-import, the check-in response time increases sharply from about 0.6 seconds to 3 seconds,
the check-out time increases accordingly from 1 to 6 seconds, then further the time gradually returns to its previous value during the entire test process.
check-in check out time chart for 5K create
at the start of the date-import 5K, delays for the CI increase from average 0.8 up to 3 sec sharply , then gradually return to its average value 0.8 sec during 5 min (all time of data import)
for CI same process with same times but values are greater of delay in 1.8 up to 5 sec than return to average during 5 min
| 5K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 3 sec | 5 min |
| Check out | 1.8 sec | 5 sec | 5 min |
check-in check out time chart for 5K update
check-in check out time chart for 10K create
| 10K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 5 sec | 11 min |
| Check out | 1.8 sec | 7 sec | 11 min |
check-in check out time chart for 10K update
check-in check out time chart for 25K create
| 25K DI | average delay | peak | return time |
|---|---|---|---|
| Check In | 0.8 sec | 3 sec | 15 min |
| Check out | 1.8 sec | 5 sec | 15 min |
heck-in check out time chart for 25K update