DATA Import Marc Authorities Testing - June 14, 2022
modules versions
Inventory Module (mod-inventory-18.1.6)
Inventory Storage Module (mod-inventory-storage-23.0.2)
Inventory Update Module (mod-inventory-update-2.0.2)
Source Record Manager Module (mod-source-record-manager-3.3.8)
Source Record Storage Module (mod-source-record-storage-5.3.2)
Data Import Module (mod-data-import-2.4.2)
Data Import Converter Storage (mod-data-import-converter-storage-1.13.3)
file | duration | time to create 1 record (sec) |
|---|---|---|
5K | 00:02:54.122 | 0.035 |
10K | 00:04:36.96 | 0.028 |
25K | 00:10:56.179 | 0.026 |
50K | 00:20:51.024 | 0.025 |
maximum number of record may be created in 30 min is 75000, without background activities (150000 in one hour)
and with maximum background activities is about only 190 records (CICO 20 users and DI on three tetants same time)
I guess the most efficient is data import authorities - long files, about 50K, since the performance is maximum and the probability of an error during download is still small.
multitenant test
test performed with file 25K all tetants run at the same time with background activities CICO 20 USERS
fs09000000 | fs07000001 | fs07000002 |
|---|---|---|
00:43:06.658 | 00:43:51.385 | 00:45:18.735 |
time to create 1 record 0.11 sec
DI with background CICO one tenant
average response time CICO + DATA Import Marc Authorities 20users 1 tenant
Check-In Controller | 1.610 |
Check-Out Controller | 1.867 |
CICO without background activities
Check-In Controller | 0.836 |
Check-Out Controller | 1.022 |
Multitenant Authority Data Import> 2 kafka partitions
file | fs09000000 | fs07000001 | fs07000002 |
|---|---|---|---|
5K | 00:07:36.579 | 00:07:39.64 | 00:08:10.303 |
10K | 00:14:50.992 | 00:12:56.94 | 00:14:21.905 |
25K | 00:40:39.063 | 00:42:00.941 | 00:42:21.044 |
50K | 01:37:59.688 | 01:38:11.591 | 01:14:52.07 |
one tenant > 2 kafka partitions
file | duration |
|---|---|
5K | 00:02:03.027 |
10K | 00:02:52.886 |
25K | 00:11:18.37 |
50K | 00:25:24.663 |
one tenant > 1 kafka partition
|
|
|---|---|
5K | 00:02:37.534 |
10K | 00:04:09.538 |
25K | 00:10:11.571 |
50K |
|