/
Data Import > Create MARC authority records with CICO

Data Import > Create MARC authority records with CICO

Feb 11, 2022

https://folio-org.atlassian.net/browse/UIDATIMP-1033

#records

began

end

Duration min

#records

began

end

Duration min

5000

8:14 AM 2/3/2022

8:16 AM 2/3/2022

2

10000

8:35 AM 2/3/2022

8:39 AM 2/3/2022

4

25000

8:42 AM 2/3/2022

8:49 AM 2/3/2022

7

50000

10:45 AM 2/3/2022

10:58 AM 2/3/2022

13

CICO activity during data import  5K 10K and 25K

CICO activity during data import  50K two times (underlined in red)

summary CICO requst  times

CI-CO 8 users

CI-CO 8 users + SRS MARC Authority Data Import

CI-CO 20 users

CI-CO 20 users + SRS MARC Authority Data Import

Comparisons

Check-in-check-out response times are affected by data import. The response time is about 2x to 3x for check in and check out.

 

8 users

8 users + 50KCreate SRS MARC Authority 

20 users

20 users + 50KCreate SRS MARC Authority 

CI average

0.618s

0.867s

0.639s

1.136s

CO average

1.22s

1.786s

1.159s

2.054s

CI max

2.078s

2.959s

1.672s

1.888s

CO max

2.340s

3.329s

2.7s

3.697s

With background activities, MARC Authority imports are slightly slower.

 

Profile

Standalone Import Durations

Data Import Durations with CICO

 

Profile

Standalone Import Durations

Data Import Durations with CICO

1K

Default - Create SRS MARC Authority

40 sec

 

5K

Default - Create SRS MARC Authority

1 min 14 sec

2 min

10K

Default - Create SRS MARC Authority

2 min 32 sec

4 min

25K

Default - Create SRS MARC Authority

6 min

7 min

50K

Default - Create SRS MARC Authority

12 min

13 min

It's clear that (MARC Authority) data import affects check-in check-out response times much more than check-in check-out affects data import durations.

max resource usage for modules comparison with Data Import MARC Authorities (Lotus Snapshot)

test

module 

CPU %

CPU % with CICO in background

50k 

mod-source-record-manager

685

955  (139%)

50k

mod-data-import-cs

404

410 (101%)

50k

okapi

385

630 (163%)

50k

mod-inventory

235

270 (114%)

50k

mod source-record-storage

231

440 (190%)

50k

mod-inventory-storage

117

380 (324%)

50k

mod-data-import

111

260 (222%)

max resource usage:

module 

RAM %

RAM % with CICO in background

mod-source-record-manager

60

47 (78%)

mod-data-import-cs

80

77 (96%)

okapi

76

65 (85%)

mod-inventory

87

56 (64%)

mod source-record-storage

78-80

61 (78%)

mod-inventory-storage

51

79 (154%)

mod-data-import

22

22 (100%)

Instance CPU during data import  50K two times

Memory during data import  50K two times after restart modules.

Instance CPU during data import  50K and CICO 20 users in background

Service CPU during data import  50K and CICO 20 users in background

Memory usage during data import  50K and CICO 20 users in background