Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Ramsons release [ECS].
Ticket: PERF-964 - Getting issue details... STATUS

Summary

All Data-imports jobs finished successfully without errors.
Duration of data imports for creates and updates are mostly the same as was in Q release.
The PTF - Updates Success - 2 profile(based on rcp1: PTF - Updates Success - 6 ) was created for the RCON Ramsons release on tenant: cs00000int_0001.
DI duration growth correlates to the number of records imported.
No memory leak is suspected for DI modules.
Services CPU utilization, Service memory utilization, and DB CPU utilization have the same utilization trend and values as in the Q release.

Recommendations & Jiras

mod-search deadlocks ticket MSEARCH-932 - Getting issue details... STATUS

Results

Test #	Data-import test	Profile	Duration Ramsons (rcon)	Duration Quesnelia (qcon)	Duration Quesnelia (qcp1)	Results
1	10k MARC BIB Create	PTF - Create 2	5 min 10 s	4 min 14 sec	6 minutes	Completed
2	25k MARC BIB Create	PTF - Create 2	10 min 30 s	9 min 41 sec	13 min 41 sec	Completed
3	50k MARC BIB Create	PTF - Create 2	15 min 43 s	18 min 18 sec	21 min 59 sec	Completed
4	100k MARC BIB Create	PTF - Create 2	31 min 51 s	38 min 36 sec	40 min 16 sec	Completed
5	500k MARC BIB Create	PTF - Create 2	2 hr 37 min	3 hours 30 min	3 hours 27 min	Completed
6	10k MARC BIB Update	PTF - Updates Success - 6	7 min 10 s	5 min 59 sec	10 min 27 sec	Completed
7	25k MARC BIB Update	PTF - Updates Success - 6	19 min 3 s	19 min 52 sec	23 min 16 sec	Completed
8	50k MARC BIB Update	PTF - Updates Success - 6	38 min 53 sec	37 min 53 sec	40 min 52 sec	Completed
9	100k MARC BIB Update	PTF - Updates Success - 6	1 hr 23 min	1 hrs 14 min	1 hrs 2 min	Completed
10	500k MARC BIB Update	PTF - Updates Success - 6		6 hrs 39 min	5 hrs 31 min	Completed

Memory Utilization

Memory usage for both sets of tests (Creates and Updates) showing stable trend. Memory of all modules returned to normal state after tests finished. No memory leak suspects observed.

Memory usage for set of MARC BIB creates.

Memory usage for set of MARC BIB updates

CPU Utilization

CPU Usage is stable for all modules involved for creates and updates. Most used module is mod-inventory (10%).

RDS Metrics

Database CPU was stable for 10 K 25K 50K 100 K and 500K records. As expected DB CPU reached 100% and stayed same during all tests.

DB metrics for create tests

Note: on 10K and 25K Create tests mod-orders was triggered. It was disabled on 50,100,500K tests. Results was not affected.

Note: here visible that mod-search queries is not included in top 10 queries, meaning that mod-search runtime indexing issue are fixed.

DB metrics for update tests

Open Search service

Open search CPU utilisation didn’t exceeds 30% on both data and master nodes, showing stable trend.

CPU usage data nodes

CPU usage master node

MSK service

MSK service showed stable trend. Max CPU usage during tests was ±60% on one of brokers.

Disk usage on all brokers didn’t exceed 10%. (300GB of memory is allocated per broker).

Additional information from module and database logs

Deadlocks observed on DB side during creates and updates data import. These deadlock is not affecting functionality of DI itself and runtime indexing functionality too as mod-search handling deadlocks on back-end side. (Deadlocks happening during runtime reindexing when mod-search working with DB). Ticket created MSEARCH-932 - Getting issue details... STATUS

Errors

No critical errors observed during data import (Creates and Updates).

The only issue observed during 500K create import. 8 records failed to create due to data in file corruption.

Appendix

Infrastructure

PTF -environment rcon

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r6.xlarge database instances, writer
MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x (KRaft mode)
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Cluster Resources - rcon-pvt

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size	R/W Split Enabled
mod-remote-storage	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.3.3	2	4920	4472	0	3960	512	512	false
mod-finance-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.7.3	2	1024	896	0	700	88	128	false
mod-ncip	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.15.6	2	1024	896	0	768	88	128	false
mod-agreements	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.1.4	2	1592	1488	0	0	0	0	false
mod-ebsconet	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.3.1	2	1248	1024	0	700	128	256	false
mod-organizations	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:2.0.0	2	1024	896	0	700	88	128	false
mod-consortia	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.2.2	2	5136	4776	0	2048	512	1024	false
edge-sip2	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.3.1	2	1024	896	0	768	88	128	false
mod-serials-management	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.1.2	2	2480	2312	0	1792	384	512	false
mod-settings	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.1.0	2	1024	896	0	768	88	128	false
mod-data-import	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.2.4	1	2048	1844	0	1292	384	512	false
mod-search	18	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:4.0.7	2	2592	2480	0	1440	512	1024	false
edge-dematic	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.3.1	1	1024	896	0	768	88	128	false
mod-inn-reach	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024	false
mod-record-specifications	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-record-specifications:1.0.2	2	1024	896	0	768	88	128	false
mod-tags	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.3.0	2	1024	896	0	768	88	128	false
mod-authtoken	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.16.1	2	1440	1152	0	922	88	128	false
edge-courses	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.5.1	2	1024	896	0	768	88	128	false
mod-notify	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.3.0	2	1024	896	0	768	88	128	false
mod-inventory-update	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:4.0.0	2	1024	896	0	768	88	128	false
mod-configuration	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.11.0	2	1024	896	0	768	88	128	false
mod-orders-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.8.3	2	1024	896	0	700	88	128	false
edge-caiasoft	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.3.2	2	1024	896	0	768	88	128	false
mod-login-saml	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.9.3	2	1024	896	0	768	88	128	false
mod-erm-usage-harvester	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:5.0.1	2	1024	896	0	768	88	128	false
mod-gobi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.9.0	2	1024	896	0	700	88	128	false
mod-licenses	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.1.2	2	2480	2312	0	1792	384	512	false
mod-password-validator	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.3.0	2	1440	1298	0	768	384	512	false
edge-dcb	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.2.1	2	1024	896	0	768	88	128	false
mod-bulk-operations	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.1.8	2	3072	2600	0	1536	384	512	false
mod-fqm-manager	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:3.0.7	2	3000	2600	0	768	88	128	false
mod-graphql	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.13.1	2	1024	896	0	768	88	128	false
mod-finance	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:5.0.1	2	1024	896	0	700	88	128	false
mod-erm-usage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:5.0.0	2	2800	2550	0	1800	384	512	false
mod-batch-print	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.2.0	2	1024	896	0	768	88	128	false
mod-tlr	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128	false
mod-lists	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:3.0.5	2	6000	2600	0	768	88	128	false
mod-copycat	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.7.0	2	1024	512	0	768	88	128	false
mod-entities-links	11	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.1.3	2	2592	2480	0	1440	0	1024	false
mod-permissions	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.6.1	2	1684	1544	512	1024	384	512	false
pub-edge	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.14	2	1024	896	0	768	0	0	false
mod-orders	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.9.9	2	2048	1740	0	1024	384	512	false
edge-patron	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.2.1	2	1024	896	0	768	88	128	false
mod-marc-migrations	26	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-marc-migrations:1.0.0	2	1024	896	0	768	88	128	false
edge-ncip	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.10.1	2	1024	896	0	768	88	128	false
edge-inn-reach	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128	false
mod-users-bl	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.9.3	2	1440	1152	0	922	88	128	false
mod-oa	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128	false
mod-inventory-storage	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:28.0.4	2	4096	3690	0	3076	512	1024	false
mod-invoice	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.9.2	2	1440	1152	0	922	88	128	false
mod-user-import	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.9.0	2	1024	896	0	768	88	128	false
mod-sender	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.13.0	2	1024	896	0	768	88	128	false
edge-oai-pmh	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.10.0	2	1512	1360	0	1440	384	512	false
mod-data-export-worker	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.3.6	2	3072	2048	0	2048	384	512	false
mod-rtac	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.7.0	2	1024	896	0	768	88	128	false
mod-circulation-storage	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.3.3	2	2880	2592	0	1814	384	512	false
mod-source-record-storage	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.9.5	2	5600	5000	0	3500	384	512	false
mod-calendar	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.2.0	2	1024	896	0	768	88	128	false
mod-event-config	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.8.0	2	1024	896	0	768	88	128	false
mod-courses	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.11	2	1024	896	0	768	88	128	false
mod-circulation-item	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.1.0	2	1024	896	0	0	0	0	false
mod-inventory	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:21.0.5	2	2880	2592	0	1814	384	512	false
mod-email	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.18.1	2	2800	2550	0	1800	384	512	false
mod-requests-mediated	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128	false
mod-circulation	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.3.8	2	2880	2592	0	1814	384	512	false
mod-pubsub	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.15.3	2	1536	1440	0	922	384	512	false
mod-di-converter-storage	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.3.1	2	1024	896	0	768	88	128	false
edge-rtac	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.8.0	2	1024	896	0	768	88	128	false
edge-orders	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.1.0	2	1024	896	0	768	88	128	false
mod-users	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.4.5	2	1024	896	0	768	88	128	false
mod-template-engine	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.21.0	2	1024	896	0	768	88	128	false
mod-patron-blocks	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.11.1	2	1024	896	0	768	88	128	false
mod-audit	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.10.2	2	1024	896	0	768	88	128	false
edge-fqm	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:3.0.2	2	1024	896	0	768	88	128	false
mod-source-record-manager	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.9.5	2	5600	5000	0	3500	384	512	false
nginx-edge	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.14	2	1024	896	0	0	0	0	false
mod-quick-marc	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512	false
nginx-okapi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14	2	1024	896	0	0	0	0	false
okapi-b	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:6.1.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.2.1	2	1024	896	0	768	88	128	false
mod-invoice-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.9.1	2	1872	1536	0	1024	384	512	false
mod-reading-room	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-reading-room:1.0.0	2	1024	896	0	768	88	128	false
mod-dcb	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.2.4	2	1024	896	0	768	88	128	false
mod-service-interaction	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.1.1	2	2048	1844	0	1290	384	512	false
mod-patron	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.2.5	2	1024	896	0	768	88	128	false
mod-data-export	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.1.5	1	2048	1844	0	0	0	0	false
mod-oai-pmh	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.14.3	2	4096	3690	0	3076	384	512	false
edge-connexion	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.3.1	2	1024	896	0	768	88	128	false
mod-notes	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:6.0.0	2	1024	896	0	952	384	512	false
mod-kb-ebsco-java	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128	false
mod-organizations-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128	false
mod-data-export-spring	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.4.3	1	2048	1844	0	1536	384	512	false
mod-login	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.12.1	2	1440	1298	0	768	384	512	false
pub-okapi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	0	768	0	0	false
edge-erm	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-erm:1.3.0	2	1024	896	0	768	88	128	false
mod-eusage-reports	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128	false

R - Ramsons release

Dataset Size:

Record type	Number of records
Instances	1 163 924
Holdings	1 348 036
Items	2 091 901

Methodology/Approach

Pre-generated files were used for DI Create job profile
- 10K, 25K, 50K, 100K and 500K files.
Run DI Create on a single tenant(cs00000int_0001) one by one with the delay with files using PTF - Create 2 profile.
Prepare files for DI Update with the Data export app, using previously imported items
Run DI Update on a single tenant(cs00000int_0001) one by one with the delay with prepared files using PTF - Update Success 2 profile

1K, 10K, 25K, 50K, 100K and 500K files.

Data-import durations were obtained from DB using SQL query

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0001_mod_source_record_manager.job_execution order by started_date desc limit 1000;

Data Import test report Ramsons [non-ECS]