[Trillium] [ECS] [Data import] Create MARC authority Records
Overview
This document presents performance testing results for Data Import of MARC Authority records using a create job profile in Trillium ECS environments (tecon).
The performance evaluation was carried out across a range of records for a single tenant: 1K, 5K, 10K, 25K, and 50K records. Additionally, we conducted a data import and parallel Check-In/Check-Out test simulating 5 virtual users to assess system behavior under concurrent operations and parallel data import on 3 tenants simultaneously.
Current ticket: https://folio-org.atlassian.net/browse/PERF-1384
Previous report: [Sunflower] [non-ECS] [Data import] Create MARC authority Records + CSP1
Summary
All tests passed successfully.
Standalone authorities imports durations improved by 30-40%
There are minimal affect of CICO on Authorities data import. CICO is affected by 10% in average during imports.
Test Runs
Test | Test conditions and short description | Status |
Test 1. | Job profile KG - Create SRS MARC Authority on nonmatches to 010 $a DUPLICATE for Q. 1k - 5k - 10k - 25k -50k with 5-minute pauses between each DI | Completed |
Test 2. | Job profile KG - Create SRS MARC Authority on nonmatches to 010 $a DUPLICATE for Q 1k - 5k - 10k - 25k -50k with 5-minute pauses between each DI Check-in/check-out: 5 Virtual users | Completed |
Test 3. | Parallel, multi-tenant Data import | Completed |
Test Results and Comparison
Test №1
Table 1. - Test with 1k, 10k, 25k, and 50k records files DI, and comparative results between Trillium and Sunflower.
Number of records | DI duration Ramsons | DI duration Sunflower | DI duration Trillium | Time Diff S vs T | Improvement S vs T |
|---|---|---|---|---|---|
1,000 | 21 sec | 16 sec | 17 sec | +1 sec | -6.25% |
5,000 | 1 min 04 sec | 1 min 03 sec | 34 sec | -29 sec | 46.03% |
10,000 | 2 min 36 sec | 2 min 03 sec | 1 min 06 sec | -57 sec | 46.34% |
25,000 | 5 min 55 sec | 4 min 39 sec | 2 min 37 sec | -2 min 02 sec | 43.73% |
50,000 | 10 min 31 sec | 9 min 09 sec | 5 min 02 sec | -4 min 07 sec | 44.99% |
Test №2
Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K, and 50K.
Table 2. - Сomparative Baseline Check-In\Check-Out results without Data Import between Trillium and Sunflower.
Data import of authorities with CICO didn’t show any degradation and stayed the same. moreover comparing with previous results from Sunflower it got improvements.
Data import affect on CheckIn-CheckOut by 26% only during 50K import. For most of cases affect is minimal.
Number of records | DI duration Sunflower with CICO | DI duration Trillium with CICO | Time Diff S vs T | Improvement S vs T | CI time T Avg, sec | CO time T Avg, sec | Baseline CI | Baseline CO |
|---|---|---|---|---|---|---|---|---|
1,000 | 17 sec | 11 sec | 6 sec | 35.29% | 417 | 951 | +3.22% | +0.32% |
5,000 | 1 min 02 sec | 39 sec | 23 sec | 37.10% | 401 | 906 | -0.74% | -4.43% |
10,000 | 1 min 59 sec | 1 min 02 sec | 57 sec | 47.90% | 431 | 954 | +6.68% | +0.63% |
25,000 | 4 min 59 sec | 2 min 34 sec | 2 min 25 sec | 48.49% | 462 | 1042 | +14.36% | +9.92% |
50,000 | 10 min 03 sec | 5 min 01 sec | 5 min 02 sec | 50.08% | 510 | 1081 | +26.24% | +14.03% |
Test №3
Table 4. - Duration on parallel multitenant data-import
Tenant | 50K DI |
|---|---|
Central - cs00000int | 10 min 44 sec |
College- cs00000int_0001 | 14 min 28 sec |
Professional- cs00000int_0002 | 14 min 46 sec |
Resource utilization for Test 1
Service CPU Utilization
The image shows CPU consumption during Test 1.
CPU usage showed predictable trends. Most used modules:
mod-source-record-storage - 922%
mod-entities-links -376%
mod-source-record-manager - 329%
Instances CPU utilization
Max CPU utilization of EC2 instance was±30% with 50K file test.
Service memory utilization
Service memory utilization remains consistent across all modules. No sudden spices or drops.
DB CPU Utilization
Here are the conclusions drawn from the database CPU usage graph:
For 1k records, the maximum CPU usage was approximately 17%.
For 5k records, the maximum CPU usage reached around 47%.
For 10k records, the maximum CPU usage climbed to about 55%.
For both 25k and 50k records, the maximum CPU usage was around 90%.
DB Connections
Max number of connections used is 979.
Database load
Sliced by SQL
Top SQL queries during test 1
no slow queries observed during tests.
Resource utilization for Test 2
CICO Response time graph
Service CPU Utilization
The image shows CPU consumption during Test 2
CPU usage for most used modules are about the same as in stand-alone test.
Service memory utilization
Service memory utilization remains consistent across related modules.
Services memory usage is steady. no sudden drops or spikes.
DB CPU Utilization
Here are the conclusions drawn from the database CPU usage graph:
DB Connections
In the idle state number of connection ~ and during CICO 5VU + 50K DI ~
Database load
Sliced by SQL
Top SQL queries during test 2
Resource utilization for Test 3
Service CPU Utilization
The image shows CPU consumption during Test 3
Most loaded module is source-record-storage ±1120%.
Service memory utilization
Service memory utilization remains consistent across all modules.
There was a spike of memory usage of mod-source-record-manager however after the end of a test memory usage is back to normal.
DB CPU Utilization
The maximum CPU usage was approximately %.
DB Connections
Max number of connections used is 1320.
Database load
Sliced by SQL
No slow queries found during tests.
Appendix
Infrastructure
PTF - environment Trillium(tecon)
tecon 8 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r7g.xlarge database instance: Writer instance
MSK fse-test
4 kafka.m7g.2xlarge brokers in 2 zones (2 brokers per zone)
Apache Kafka version 3.7.x, metadata mode - KRaft
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
revision - 26
OpenSearch 2.13 ptf-test cluster
r7g.2xlarge.search 4 data nodes
r7g.large.search 3 dedicated master nodes
Kafka consolidated topics enabled