Overview The purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in scope of Trillium regression ECS testing https://folio-org.atlassian.net/browse/PERF-1358
Previous report https://folio-org.atlassian.net/wiki/x/AQByE
Summary All tests completed successfully.
Most loaded modules are mod-rtac-cache - 468%, mod-source-record-storage - 335% , mod-search - 285%. There’s an issue with mod-rtac-cache it’s spiking during DI (ticket for anomaly resource usage created)
Durations for all file sizes is 35-50% lower than in previous report(Quesnelia). There’s visible improvements on DB side Data Import Create MARC Holdings Records [ECS] [Quesnelia] Data Import Create MARC Holdings Records [ECS] [Trillium]
No memory leaks or additional anomalies found during tests (Except mod-rtac-cache issue)
Recommendations & Jiras Test Runs/Results Profile used for testing - Default - Create Holdings and SRS MARC Holdings
Test Results
1
1k
1 min 3 sec
34 s
-46.03%
Success
2
5k
4 min 16 sec
1 min 54 s
-55.47%
Success
3
10k
8 min 59 sec
3 min 56 s
-56.22%
Success
4
80k
52 min 5 sec
34 min 20 s
-34.08%
Success
Cluster resource utilization Service CPU Utilization Most used modules (CPU %):
Note: All CPU trends looks normal, showing predictable behavior during DI. It’s clearly visible on this screen - mod-rtac-cahce keep being loaded even after 80K test is done for almost 40 min.
Memory Utilization Memory utilization for all related modules stayed stable, without spikes and anomalies.
Instance CPU utilization Maximum loaded EC2 instance has 25%. which mean that there's enough room for other workflows.
MSK tenant cluster Disk usage by broker CPU (User) usage by broker DB resources utilization RDS CPU Utilization DB Connections
DB Load
SQL queries Infrastructure PTF - environment Trillium(tecon)
cs00001
1875246
22
25
cs00001_0001
7177106
7102198
7795066
cs00001_0002
250656
423843
845449
cs00001_0003
252910
430519
856456
cs00001_0004
233675
263948
611709
cs00001_0005
436241
790574
1653178
tecon cluster resources Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 1 mod-remote-storage:4.0.0 2 4920 4472 64 3960 512 512 mod-remote-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-finance-storage 1 mod-finance-storage:8.9.1 2 1024 896 64 700 88 128 mod-finance-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-mosaic 1 mod-mosaic:1.1.0 2 1024 896 64 768 88 128 mod-mosaic - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-ebsconet 1 mod-ebsconet:2.5.0 2 1248 1024 64 700 128 256 mod-ebsconet - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-sip2 2 edge-sip2:4.0.2 2 1024 896 64 768 88 128 mod-consortia-keycloak 1 mod-consortia-keycloak:1.8.1 2 5136 4776 512 4416 384 512 mod-consortia-keycloak - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-tags 1 mod-tags:3.0.0 2 1024 896 64 768 88 128 mod-tags - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-courses 1 edge-courses:1.7.0 2 1024 896 64 768 88 128 mod-notify 1 mod-notify:3.5.0 2 1024 896 64 768 88 128 mod-notify - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-inventory-update 3 mod-inventory-update:5.1.3 2 1024 896 64 768 88 128 mod-inventory-update - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-configuration 1 mod-configuration:5.13.0 2 1024 896 64 768 88 128 mod-configuration - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-caiasoft 1 edge-caiasoft:3.0.0 2 1024 896 64 768 88 128 folio-keycloak 4 folio-keycloak:26.6.0 3 2750 2250 1536 1536 0 0 mod-licenses 2 mod-licenses:6.3.3 2 2480 2312 64 1792 384 512 mod-licenses - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-gobi 1 mod-gobi:3.1.0 2 1024 896 64 768 88 128 mod-gobi - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-graphql 2 mod-graphql:1.15.1 2 1024 896 64 768 88 128 mod-graphql - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-roles-keycloak 1 mod-roles-keycloak:4.0.3 4 4096 3688 128 3076 88 256 mod-roles-keycloak - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 128 256 0 96 mod-erm-usage 1 mod-erm-usage:5.2.0 2 2800 2550 64 1800 384 512 mod-erm-usage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-batch-print 1 mod-batch-print:1.4.0 2 1024 896 64 768 88 128 mod-batch-print - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-copycat 1 mod-copycat:1.9.0 2 1024 896 64 768 88 128 mod-copycat - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-entities-links 1 mod-entities-links:5.0.2 2 2592 2483 64 1440 0 1024 mod-entities-links - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 pub-edge 2 pub-edge:2025.09.19 2 1024 896 64 768 0 0 mod-orders 2 mod-orders:13.1.6 2 2048 1740 128 1024 384 512 mod-orders - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 kong 5 kong:3.9.6 2 6144 6144 64 768 88 128 edge-ncip 2 edge-ncip:1.11.3 2 1024 896 64 768 88 128 mod-users-bl 1 mod-users-bl:8.0.0 2 1440 1152 64 922 88 128 mod-users-bl - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-invoice 1 mod-invoice:6.1.3 2 1440 1152 64 922 88 128 mod-invoice - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-inventory-storage 3 mod-inventory-storage:30.0.3 2 4300 3690 512 3067 384 512 mod-inventory-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 768 512 256 0 128 mod-user-import 1 mod-user-import:3.11.0 2 1024 896 64 768 88 128 mod-user-import - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-sender 1 mod-sender:1.15.0 2 1024 896 64 768 88 128 mod-sender - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mgr-tenant-entitlements 4 mgr-tenant-entitlements:4.0.1 2 1024 896 64 512 88 256 mod-login-keycloak 1 mod-login-keycloak:4.0.1 2 1024 896 128 512 88 256 mod-login-keycloak - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-circulation-storage 1 mod-circulation-storage:17.5.0 2 2880 2592 64 1814 384 512 mod-circulation-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-calendar 1 mod-calendar:4.0.0 2 2048 1536 64 1024 96 256 mod-calendar - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-event-config 1 mod-event-config:2.10.0 2 1024 896 64 768 88 128 mod-event-config - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-courses 1 mod-courses:1.6.0 2 1024 896 64 768 88 128 mod-courses - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-circulation-item 1 mod-circulation-item:2.0.0 2 1024 896 64 0 0 0 mod-circulation-item - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-email 1 mod-email:1.20.0 2 2800 2550 64 1800 384 512 mod-email - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-di-converter-storage 1 mod-di-converter-storage:2.5.0 2 1024 896 64 768 88 128 mod-di-converter-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-circulation 1 mod-circulation:24.5.0 2 2880 2592 64 1814 384 512 mod-circulation - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-pubsub 1 mod-pubsub:2.17.0 2 1536 1440 1024 922 384 512 mod-pubsub - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 512 256 0 96 edge-rtac 1 edge-rtac:3.1.0 2 1024 896 64 768 88 128 edge-orders 1 edge-orders:3.3.0 2 1024 896 64 768 88 128 mod-linked-data-import 1 mod-linked-data-import:1.0.1 2 2048 1792 64 768 88 128 mod-linked-data-import - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-rtac-cache 6 mod-rtac-cache:1.1.1 2 1024 896 64 768 88 128 mod-rtac-cache - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-template-engine 1 mod-template-engine:1.23.0 2 1024 896 64 768 88 128 mod-template-engine - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-users 1 mod-users:19.6.0 2 1024 896 64 768 88 128 mod-users - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-audit 2 mod-audit:3.0.1 2 1024 896 64 768 88 128 mod-audit - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-source-record-manager 2 mod-source-record-manager:4.0.5 2 4500 4000 64 3000 384 512 mod-source-record-manager - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-quick-marc 1 mod-quick-marc:8.0.1 1 2288 2176 64 1664 384 512 mod-quick-marc - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-feesfines 1 mod-feesfines:19.4.0 2 1024 896 64 768 88 128 mod-feesfines - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-users 1 edge-users:2.2.0 2 1024 896 64 768 88 128 mod-service-interaction 2 mod-service-interaction:4.3.2 2 2048 1844 64 1290 384 512 mod-service-interaction - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-dcb 1 mod-dcb:2.0.0 2 2048 1844 64 768 88 192 mod-dcb - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-patron 2 mod-patron:6.4.1 2 1024 896 64 768 88 128 mod-patron - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-connexion 1 edge-connexion:1.5.0 2 1024 896 64 768 88 128 mod-organizations-storage 1 mod-organizations-storage:5.0.0 2 1024 896 64 620 88 128 mod-organizations-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-data-export-spring 1 mod-data-export-spring:3.6.1 1 2048 1844 64 1536 384 512 mod-data-export-spring - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-erm 2 edge-erm:1.5.0 2 1024 896 64 768 88 128 mod-ncip 1 mod-ncip:1.17.3 2 1024 896 64 768 88 128 mod-ncip - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-agreements 2 mod-agreements:7.3.4 2 4096 4096 64 0 0 0 mod-agreements - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-organizations 1 mod-organizations:2.2.0 2 1024 896 64 768 88 128 mod-organizations - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-serials-management 2 mod-serials-management:2.1.2 2 2780 2312 64 1792 384 896 mod-serials-management - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-settings 1 mod-settings:1.3.1 2 1024 896 64 768 88 128 mod-settings - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-data-import 2 mod-data-import:3.4.0 1 2048 1844 64 1292 384 512 mod-data-import - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-search 2 mod-search:6.0.6 2 2592 2480 64 1440 512 1024 mod-search - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 128 256 0 96 edge-dematic 1 edge-dematic:3.0.0 1 1024 896 64 768 88 128 mod-record-specifications 1 mod-record-specifications:3.0.0 2 2048 1592 64 1440 256 512 mod-record-specifications - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-inventory 1 edge-inventory:2.3.0 2 1024 896 64 768 88 128 mod-orders-storage 1 mod-orders-storage:14.0.2 2 1024 896 64 700 88 128 mod-orders-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-erm-usage-harvester 1 mod-erm-usage-harvester:5.2.0 2 1024 896 64 768 88 128 mod-erm-usage-harvester - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-password-validator 1 mod-password-validator:4.0.0 2 1440 1298 64 768 384 512 mod-password-validator - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-bulk-operations 2 mod-bulk-operations:2.3.4 2 5600 5200 64 2880 384 512 mod-bulk-operations - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1500 768 64 256 0 128 mod-fqm-manager 2 mod-fqm-manager:4.1.6 2 3000 2600 64 2048 384 512 mod-fqm-manager - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-dcb 1 edge-dcb:2.0.0 2 1024 896 64 768 88 128 mod-finance 1 mod-finance:5.2.1 2 1024 896 64 768 88 128 mod-finance - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-lists 2 mod-lists:4.0.1 2 6000 2600 64 2048 256 512 mod-lists - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-permissions 1 mod-permissions:6.8.0 2 1684 1544 64 1024 384 512 mod-permissions - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-users-keycloak 1 mod-users-keycloak:4.0.2 4 4096 3688 128 3076 88 256 mod-users-keycloak - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 128 256 0 96 mod-marc-migrations 3 mod-marc-migrations:3.0.2 2 2880 2592 64 1814 384 512 mod-marc-migrations - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mgr-applications 2 mgr-applications:4.0.1 2 1024 896 64 512 88 256 mgr-tenants 2 mgr-tenants:4.0.0 2 1024 896 64 512 88 256 edge-oai-pmh 1 edge-oai-pmh:2.12.0 2 1512 1360 64 1440 384 512 mod-okapi-facade 1 mod-okapi-facade:4.0.0 2 1024 896 64 768 88 128 mod-okapi-facade - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-rtac 1 mod-rtac:3.9.1 2 1024 896 64 768 88 128 mod-rtac - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-scheduler 2 mod-scheduler:4.0.2 2 1024 896 64 512 88 256 mod-scheduler - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-source-record-storage 2 mod-source-record-storage:6.0.1 2 4500 4000 64 3000 384 512 mod-source-record-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-inventory 4 mod-inventory:22.0.2 2 4096 3688 512 3076 384 512 mod-inventory - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 768 512 256 0 128 mod-linked-data 2 mod-linked-data:2.0.3 0 2048 1792 64 768 88 128 mod-linked-data - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-patron-blocks 1 mod-patron-blocks:1.13.0 2 1024 896 64 768 88 128 mod-patron-blocks - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 edge-fqm 2 edge-fqm:4.1.2 2 1024 896 64 768 88 128 mod-reporting 1 mod-reporting:1.6.1 2 1024 896 64 768 88 128 mod-reporting - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 nginx-edge 2 nginx-edge:2025.09.19 2 1024 896 64 0 0 0 mod-invoice-storage 1 mod-invoice-storage:6.1.1 2 1872 1536 64 1024 384 512 mod-invoice-storage - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-reading-room 1 mod-reading-room:2.0.0 2 1024 896 64 768 88 128 mod-reading-room - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-data-export 1 mod-data-export:5.4.4 1 2592 2480 64 1440 88 1024 mod-data-export - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-oai-pmh 1 mod-oai-pmh:3.16.2 2 4096 3690 64 3076 384 512 mod-oai-pmh - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-notes 1 mod-notes:8.0.0 2 1024 896 64 952 384 512 mod-notes - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-kb-ebsco-java 2 mod-kb-ebsco-java:7.0.1 2 1024 896 64 768 88 128 mod-kb-ebsco-java - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96 mod-eusage-reports 1 mod-eusage-reports:3.1.0 2 1024 896 64 768 88 128 mod-eusage-reports - Sidecar 1 N/A folio-module-sidecar:4.0.1.nb N/A 1024 512 64 256 0 96
Methodology/Approach Prepare Data Import Files 1k, 5k, 10k, 80k with defined number of holding records associated with instance HRID (1 instance HRID for all records or 1 per 1000 records)
run Data imports using default SRS profile on 001 (College tenant )
If everything is successful - collect durations of DI with :
select file_name,total_records_in_file,started_date,completed_date, completed_date - started_date as duration ,status,error_status
from [tenant]_mod_source_record_manager.job_execution
where subordination_type = 'COMPOSITE_PARENT'
-- where started_date > '2024-06-13 14:47:54' and completed_date < '2024-06-13 19:01:50.832'
order by started_date desc
limit 10 