ECS [Trillium] PTF - Test Authority MARC Migration on Central Tenant
Overview
This document contains the results of testing marc-migration for over 12 million records on Trillium ECS environment in central tenant.
Ticket: 
Summary
Authority MARC migration competed successfully.
The duration for marc-migration is 9 hours 41 minutes (7 hours 37 min for mapping, 2 hours 4 min for saving ) for default configurations with triggering the migration using jmeter script.
Parameters for CHUNK_FETCH_IDS_COUNT=500 and RECORDS_CHUNK_SIZE=500
Service Memory utilization for mod-entities-links was in range of 34%-42%, mod-search - 40%-52%, and mod-marc-migrations - 34%
Service CPU utilization for mod-entities-links was in range of 280%-380%, mod-search - 30% while mapping and 470% while saving, and mod-marc-migrations - 170% while mapping and 13% while saving
DB CPU usage - 20% while mapping and 40% while saving
Significant degradation observed comparing to Ramsons release test with the same parameters - twice slower total duration for mapping and saving processes.
Additional testing on Sunflower cluster with 12.1 mln authorities in central tenant
2 hours 26 min for mapping, 2 hours 41 min for saving
Mapping Service CPU: mod-marc-migrations-b - 740%, DB CPU - 24%
Saving Service CPU: mod-entities-links-b - 2500%, mod-source-record-storage-b - 133%, mod-search-b - 114%, DB CPU - 80%
mod-marc-migrations:2.0.5Sunflower /mod-marc-migrations:3.0.0-SNAPSHOT.32Trillium
Update:
In scope of
mod-marc-migrations:3.1.0-SNAPSHOT.38with additional parameters:DB_MAXPOOLSIZE-remove
DB_CONNECTIONRELEASEDELAY=600000
DB_MAXSHAREDPOOLSIZE=20
DB_MINSHAREDPOOLSIZE=10
Results of retesting:
mapping duration improved from 7h 37 m to 1h 54 m / 3.99× faster
saving duration improved from 2h 04m to 28m 19 s / 4.38× faster
Phase | Duration |
|---|---|
Mapping | 1h 54m 39s |
Data saving | 28m 19s |
Effective total duration | 2h 22m |
Recommendations and Jiras
To proceed investigation of significant slowness in Trillium release compared to results from Ramsons (ECS) central tenant
Ticket:
Test № | status | total_num_of_records | mapped_num_of_records | saved_num_of_records |
|---|---|---|---|---|
Test №1 | data_mapping_completed / data_saving_completed | 12118339 | 12118339 | 12118339 |
Test Results
This table contains duration time for Migrated and saved Marc Authority records in central tenant.
Test № | CHUNK_FETCH_IDS_COUNT | RECORDS_CHUNK_SIZE | mapping_duration | saving_duration | total_duration |
|---|---|---|---|---|---|
Test №1 | 500 | 500 | 7 hours 37 min | 2 hours 4 min | 9 hours 41 min |
Comparison
Compared to the Ramsons release https://folio-org.atlassian.net/wiki/x/AYByI
Test № | CHUNK_FETCH_IDS_COUNT | RECORDS_CHUNK_SIZE | mapping_duration (Ramsons) 12067250 | mapping_duration (Trillium) 12118339 | saving_duration (Ramsons) 12067250 | saving_duration (Trillium) 12118339 | mapping_duration, delta | saving_duration, delta |
|---|---|---|---|---|---|---|---|---|
Test №1 | 500 | 500 | 3 hours 1 min | 7 hours 37 min | 1 hours 18 min | 2 hours 4 min | 4 hours 36 min (grew + 150%) | 46 min (grew +59%) |
Instance CPU Utilization
Service CPU Utilization
Service Memory Utilization
Kafka metrics
OpenSearch Data Nodes metrics
DB CPU Utilization
During mapping DB CPU - 20%, saving - 40%
DB Connections
DB connection drop from 1480 to 1080 before saving process explained by mod-serials-management service stopped (tasks count - 0) because of continuous container restarts
DB load
Top SQL-queries
Appendix
Infrastructure
PTF - environment Trillium ECS (asptc) central tenant
9 r7i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r7g.xlarge database instance: Writer instance
MSK ptf-loc-1
6 m7g.large brokers in 2 zones
Apache Kafka version 3.9.x
EBS storage volume per broker 300 GiB
OpenSearch 2.19 ptf-loc cluster
r7g.2xlarge.search 4 data nodes
m7g.large.search 3 dedicated master nodes
Central tenant:
Instance count - 20993574, Items, holdings - 0