Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device.
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
/
[Trillium] [non-ECS] [Data import] Create MARC authority Records
This document presents performance testing results for Data Import of MARC Authority records using a create job profile in the Trillium Eureka non-ECS environments (tecp1).
File size used: 1K, 5K, 10K, 25K, and 50K records. Data import tests were carried out on a single tenant without and with Check-In/Check-Out 5 virtual users. Additionally parallel data imports performed on 3 tenants with 25k and 50k records without Check-In/Check-Out. Ticket: https://folio-org.atlassian.net/browse/PERF-1389
All imports completed without failures; improved in Trillium by 30% in average
Data Import with Check-In/Check-Out concurrently (Test 2):
DI durations remained comparable to Test 1: 25K in 3m 56s (−21% vs Sunflower), 50K in 7m 36s (−24% vs Sunflower).
CICO had minimal impact on DI.
Check-In/Check-Out (avg times during DI):
CI avg: 437–764 ms (−59% to −43% vs Sunflower depending on file size)
CO avg: 896–1,262 ms (−28% to −3% vs Sunflower)
Response time degradation under DI load is gradual — CI increases from 437 ms (1K) to 764 ms (50K); CO from 896 ms to 1,262 ms.
CICO without DI:
CI: avg 431 ms (−48% vs Sunflower's 835 ms avg)
CO: avg 874 ms (−22% vs Sunflower's 1,115 ms avg)
Resource Utilization:
Service CPU: Top consumers during DI: mod-source-record-storage (~704%), mod-entities-links (~301%), mod-source-record-manager (~286%). CPU spikes correlate with larger file imports (25K, 50K).
Service Memory: Stable throughout all tests, no memory spikes or OOM. Top consumers: mgr-applications (~64%), mgr-tenant-entitlements (~58%), mod-data-export-worker (~56%), folio-keycloak (~48%), mod-search (~40%), mod-inventory (~38%).
DB (RDS) CPU: Spikes to ~96% during 10K+ record imports. Baseline idle at ~15–24%. During parallel multi-tenant Test 3, DB CPU sustained at ~96%.
DB Connections: Baseline ~693; rose to ~800–1,000 during single-tenant tests; peaked at ~1,390 during parallel multi-tenant imports.
DB Load: Peak Average Active Sessions (AAS) up to ~49. Top SQL by load: INSERT INTO marc_records_lb (AAS 2.03, 19.76 calls/sec, 103.52 ms avg latency), COMMIT (AAS 1.73), INSERT INTO journal_records (AAS 1.21, 98.86 calls/sec, 11.31 ms), INSERT INTO events_processed (AAS 0.70, 19.78 calls/sec, 6.86 ms).
Recommendations & Jiras
There’s a possible issue while importing records where field 667 has double $a parameter. Script to prepare files pushed to Git
Test Runs
Test
Test conditions and short description
Status
Test 1.
Tenant: fs0900000. Job profile KG - Create SRS MARC Authority on nonmatches to 010 $a DUPLICATE for Q. 1k - 5k - 10k - 25k -50k with 5 minutes pauses between each DI
Completed
Test 2.
Tenant: fs0900000. Job profile KG - Create SRS MARC Authority on nonmatches to 010 $a DUPLICATE for Q 1k - 5k - 10k - 25k -50k with 5 minutes pauses between each DI
CheckIn-CheckOut 5 Virtual users
Completed
Test 3.
Parallel, multi-tenant Data import Job profile KG - Create SRS MARC Authority on nonmatches to 010 $a DUPLICATE for Q. In parallel Tenant: fs0900000. 50k and 25k without pause. Tenant: fs0700001. 50k and 25k without pause. Tenant: fs0700002. 50k and 25k without pause.
Completed
Test Results and Comparison
Table contains Data import comparison between Trillium and Sunflower releases.
Number of records
DI duration Poppy
DI duration Quesnelia
DI duration Ramsons
DI duration Sunflower
DI duration Trillium
Delta, Trillium vs Sunflower
Number of records
DI duration Poppy
DI duration Quesnelia
DI duration Ramsons
DI duration Sunflower
DI duration Trillium
Delta, Trillium vs Sunflower
1,000
00:00:29
00:00:22
00:00:21
00:00:16
00:00:27
+00:00:11 / +69%
5,000
00:01:38
00:01:19
00:01:04
00:01:03
00:01:03
00:00:00 / 0%
10,000
00:02:53
00:02:36
00:02:36
00:02:03
00:01:49
−00:00:14 / −11%
25,000
00:05:42
00:06:24
00:05:55
00:04:39
00:03:47
−00:00:52 / −19%
50,000
00:13:48
00:11:59
00:10:31
00:09:09
00:06:58
−00:02:11 / −24%
Table contains results of CICO without DI
CICO, Avg time (Sunflower), ms
CICO, 95% (Sunflower), ms
CICO, Avg time (Trillium), ms
CICO, 95% time (Trillium), ms
CICO, Avg time (Sunflower), ms
CICO, 95% (Sunflower), ms
CICO, Avg time (Trillium), ms
CICO, 95% time (Trillium), ms
Check-In
835
934
431
570
Check-Out
1115
1323
874
1122
Сomparative table for data import durations and CICO response times with and without data import
Number of records
DI duration Sunflower
DI duration Trillium
DI duration delta, T/S
CI Avg Sunflower, ms
CI Avg Trillium, ms
CI delta, T/S
CO Avg Sunflower, ms
CO Avg Trillium, ms
CO delta, T/S
Number of records
DI duration Sunflower
DI duration Trillium
DI duration delta, T/S
CI Avg Sunflower, ms
CI Avg Trillium, ms
CI delta, T/S
CO Avg Sunflower, ms
CO Avg Trillium, ms
CO delta, T/S
1,000
00:00:17
00:00:15
−00:00:02 / −12%
1,064
437
−627 / −59%
1,238
896
−342 / −28%
5,000
00:01:02
00:01:03
+00:00:01 / +2%
906
523
−383 / −42%
1,242
1,005
−237 / −19%
10,000
00:01:59
00:01:57
−00:00:02 / −2%
974
684
−290 / −30%
1,239
1,198
−41 / −3%
25,000
00:04:59
00:03:56
−00:01:03 / −21%
1,105
692
−413 / −37%
1,368
1,190
−178 / −13%
50,000
00:10:03
00:07:36
−00:02:27 / −24%
1,331
764
−567 / −43%
1,548
1,262
−286 / −18%
Test scenario
CI Avg Sunflower, ms
CI Avg Trillium, ms
CI Avg delta, T/S
CI 95% Sunflower, ms
CI 95% Trillium, ms
CO Avg Sunflower, ms
CO Avg Trillium, ms
CO Avg delta, T/S
CO 95% Sunflower, ms
CO 95% Trillium, ms
CICO without DI
835
431
−404 / −48%
934
570
1,115
874
−241 / −22%
1,323
1,122
Parallel data-import tests on 3 tenants in Trillium