DATA Import Marc Authorities Testing - June 14, 2022

UIDATIMP-1128 - Getting issue details... STATUS

modules versions

Inventory Module (mod-inventory-18.1.6)
Inventory Storage Module (mod-inventory-storage-23.0.2)
Inventory Update Module (mod-inventory-update-2.0.2)
Source Record Manager Module (mod-source-record-manager-3.3.8)
Source Record Storage Module (mod-source-record-storage-5.3.2)
Data Import Module (mod-data-import-2.4.2)
Data Import Converter Storage (mod-data-import-converter-storage-1.13.3)

file	duration	time to create 1 record (sec)
5K	00:02:54.122	0.035
10K	00:04:36.96	0.028
25K	00:10:56.179	0.026
50K	00:20:51.024	0.025

maximum number of record may be created in 30 min is 75000, without background activities (150000 in one hour)

and with maximum background activities is about only 190 records (CICO 20 users and DI on three tetants same time)

I guess the most efficient is data import authorities - long files, about 50K, since the performance is maximum and the probability of an error during download is still small.

multitenant test

test performed with file 25K all tetants run at the same time with background activities CICO 20 USERS

fs09000000	fs07000001	fs07000002
00:43:06.658	00:43:51.385	00:45:18.735

time to create 1 record 0.11 sec

DI with background CICO one tenant

average response time CICO + DATA Import Marc Authorities 20users 1 tenant

Check-In Controller	1.610
Check-Out Controller	1.867

CICO without background activities

Check-In Controller	0.836
Check-Out Controller	1.022

Multitenant Authority Data Import> 2 kafka partitions

file	fs09000000	fs07000001	fs07000002
5K	00:07:36.579	00:07:39.64	00:08:10.303
10K	00:14:50.992	00:12:56.94	00:14:21.905
25K	00:40:39.063	00:42:00.941	00:42:21.044
50K	01:37:59.688	01:38:11.591	01:14:52.07

one tenant > 2 kafka partitions

file	duration
5K	00:02:03.027
10K	00:02:52.886
25K	00:11:18.37
50K	00:25:24.663

one tenant > 1 kafka partition


5K	00:02:37.534
10K	00:04:09.538
25K	00:10:11.571
50K