Data Import with Check-ins Check-outs
- Iakov Zaborov
- Martin Tran
Infrastructure
- 6 m5.xlarge EC2 instances
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- mod-inventory memory
- 256 CPU units, 1814MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 128 CPU units, 544MB mem
- mod-source-record-storage
- 128 CPU units, 908MB mem
- mod-source-record-manager
- 128 CPU units, 1292MB mem
- mod-data-import
- 128 CPU units, 1024MB mem
Software versions
- mod-data-import v2.4.1
- mod-data-import-converter-storage v1.13.2
- mod-source-record-manager v3.3.8
- mod-source-record-storage v5.3.2
- mod-inventory v18.1.4
- mod-inventory-storage v23.0.2
Results
| MARC File | Profile | Tenant | DI Duration | Check In (CI), Check Out (CO) | |||
|---|---|---|---|---|---|---|---|
| users | tenant | CI Average (seconds) | CO Average (seconds) | ||||
| 8 | fs09000000 | 0.704 | 1.352 | ||||
| 25K.mrc | PTF - Create 2 | cap1 | 00:26:59.84 | 0 | - | ||
| 25K.mrc | PTF - Create 2 | cap1 | 00:22:37.093 | 8 | fs09000000 | 1.480 | 2.068 |
| 25K.mrc | PTF - Create 2 | cap1 | 00:34:53.31 | 20 | fs09000000 | 3.798 | 6.254 |
| 25K.mrc | PTF - Create 2 | cap1 | 00:25:58.258 | 20 | fs09000000 | 1.255 | 2.074 |
| 25K_mark_autority.mrc | Default - Create SRS MARC Authority | cap1 | 00:12:26.71 | 0 | - | - | - |
| 25K_mark_autority.mrc | Default - Create SRS MARC Authority | cap1 | 00:15:14.125 | 8 | fs09000000 | 0.784 | 1.248 |
| 25K_mark_autority.mrc | Default - Create SRS MARC Authority | cap1 | 00:15:09.881 | 20 | fs09000000 | 2.295 | 4.281 |
Comparisons
The following table compares additional test results to previous release numbers and to the CICO baselines (of Check In average time 0.704s and Checkout average time 1.352s). Note that Lotus numbers are in red, Kiwi numbers are in blue.
in the new version, there is no change in the performance of date-import, but noticeably faster check-in-checkout
Profile | Duration KIWI (Lotus) without CICO | Duration with CICO 8 users | CheckIn average (seconds) | CheckOut average (seconds) | Deviation From the baseline CICO response times | |
|---|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 5 min, 8 min (05:32.264) (08:48.556) | 5 min (05:48.671) | 1.297 0.851 | 2.449 1.388 | CI: 17% CO: 3% |
5K MARC Update | PTF - Updates Success - 1 | 11 min, 13 min (10:07.723) | 7 min 06:27.143 | 1.315 1.102 | 2.469 1.867 | CI: 56% CO: 38% |
10K MARC Create | PTF - Create 2 | 11 min , 14 min 10:07.723 | 13 min 00:10:47.622 | 1.407 1.354 | 2.795 2.271 | CI: 92% CO: 68% |
| 10K MARC Update | PTF - Updates Success - 1 | 22 min, 24 min | 15 min 20:46.29 | 1.691 0.864 | 3.041 1.655 | CI: 51% CO: 54% |
| 25K MARC Create | PTF - Create 2 | 23 mins, 25 mins, 26 mins | 31 min 32:21.273 | 1.536 1.248 | 2.962 2.078 | CI: 77% CO: 54% |
| 25K MARC Update | PTF - Updates Success - 1 | 27 min, 40 mins, 56mins | 33 min 31:40.429 | 1.589 0.951 | 2.997 1.658 | CI: 35% CO: 23% |
| 50K MARC Create | PTF - Create 2 | 40 mins, 43mins 49:55.86 | 63 min 47:53.93 | 1.783 1.271 | 3.231 2.095 | CI: 81% CO: 55% |
| 50K MARC Update | PTF - Updates Success - 1 | 64 min 48:46.35 | FAILED | - | - | CI: CO: |
5k CREATE
5K marc update
10k marc create
Lotus 25K MARC Create
Lotus 25K MARC Update
Lotus 50 k marc create
