Data Import Test Report OL enabled + background activities
Overview
This document contains the results of testing Data Import in Kiwi with OL enabled + background activities (Check-in/Check-out test with 8 users).
Infrastructure
6 m5.xlarge EC2 instances
2 instances of db.r6.xlarge database instances, one reader and one writer
MSK
4 m5.2xlarge brokers in 2 zones
auto.create-topics.enable = true
log.retention.minutes=120
mod-inventory memory
256 CPU units, 1814MB mem
inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
kafka.consumer.max.poll.records=10
mod-inventory-storage
128 CPU units, 544MB mem
mod-source-record-storage
128 CPU units, 908MB mem
mod-source-record-manager
128 CPU units, 1292MB mem
mod-data-import
128 CPU units, 1024MB mem
Software versions
mod-data-import:2.2.0
mod-inventory:18.0.4
mod-inventory-storage:22.0.2-optimistic-locking.559
mod-source-record-storage:5.2.5
mod-source-record-manager:3.2.6
Results
Tests performed:
| Profile | Duration KIWI WO background activities | Duration | Began UTC | CheckIn average | CheckOut average | Link to CheckIn-CheckOut chart |
|---|---|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 5 min, 8 min (05:32.264) | 5 min | 10:42 AM 022.01.10 | 1.297 | 2.449 | |
5K MARC Update | PTF - Updates Success - 1 | 11 min, 13 min | 7 min | 11:20 AM 2022.01.10 | 1.315 | 2.469 | |
10K MARC Create | PTF - Create 2 | 11 min , 14 min | 13 min | 2:44 PM 2022.01.10 | 1.407 | 2.795 | |
10K MARC Update | PTF - Updates Success - 1 | 22 min, 24 min | 15 min | 3:26 PM 2022.01.10 | 1.691 | 3.041 | |
25K MARC Create | PTF - Create 2 | 23 mins, 25 mins, 26 mins | 31 min | 8:55 AM 2022.01.11 | 1.536 | 2.962 | |
25K MARC Update | PTF - Updates Success - 1 | 27 min, 40 mins, 56mins | 33 min | 9:56 AM 2022.01.11 | 1.589 | 2.997 | |
50K MARC Create | PTF - Create 2 | 40 mins, 43mins, | 63 min | 12:45 PM 2022.01.12 | 1.783 | 3.231 | |
50K MARC Update | PTF - Updates Success - 1 | 64 min | FAILED | 3:56 PM 2022.01.12 | - | - |
|
Вelow are the general timings for check-in check-out. The horizontal axis shows the number of mrs records in thousands
We can see her an exponential increase of the import time depending on the file size
also there a slight linear increase in the average latency for the check-in check-out
It can be seen from the graphs below that - after the start of the data-import, the check-in response time increases sharply from about 0.6 seconds to 3 seconds,
the check-out time increases accordingly from 1 to 6 seconds, then further the time gradually returns to its previous value during the entire test process.
check-in check out time chart for 5K create
at the start of the date-import 5K, delays for the CI increase from average 0.8 up to 3 sec sharply , then gradually return to its average value 0.8 sec during 5 min (all time of data import)