1 Overview
2 Summary
- 2.1 Recommendations & Jiras
3 Results
- 3.1 Memory Utilization
- 3.2 CPU Utilization
- 3.3 RDS Metrics
- 3.4 Open Search service
- 3.5 MSK service
- 3.6 Additional information from module and database logs
4 Errors
5 Appendix
- 5.1 Infrastructure
- 5.2 Cluster Resources - rcon-pvt
- 5.3 Methodology/Approach

Overview

This document contains the results of testing Data Import for MARC Bibliographic records creates and updates with different file sizes (10K, 25K, 50K, 100K, 500K) at Ramsons release Okapi [ECS].
Ticket: https://folio-org.atlassian.net/browse/PERF-964

Summary

All Data-imports jobs finished successfully without errors.
Duration of data imports for creates and updates are mostly the same as was in Q release.
The PTF - Updates Success - 2 profile(based on rcp1: PTF - Updates Success - 6 ) was created for the RCON Ramsons release on tenant: cs00000int_0001.
DI duration growth correlates to the number of records imported.
No memory leak is suspected for DI modules.
Services CPU utilization, Service memory utilization, and DB CPU utilization have the same utilization trend and values as in the Q release.

Recommendations & Jiras

mod-search deadlocks tickethttps://folio-org.atlassian.net/browse/MSEARCH-932

Results

Test #	Data-import test	Profile	Duration Ramsons (rcon)	Duration Quesnelia (qcon)	Duration Quesnelia (qcp1)	Results

Test #	Data-import test	Profile	Duration Ramsons (rcon)	Duration Quesnelia (qcon)	Duration Quesnelia (qcp1)	Results
1	10k MARC BIB Create	PTF - Create 2	5 min 10 s	4 min 14 sec	6 minutes	Completed
2	25k MARC BIB Create	PTF - Create 2	10 min 30 s	9 min 41 sec	13 min 41 sec	Completed
3	50k MARC BIB Create	PTF - Create 2	15 min 43 s	18 min 18 sec	21 min 59 sec	Completed
4	100k MARC BIB Create	PTF - Create 2	31 min 51 s	38 min 36 sec	40 min 16 sec	Completed
5	500k MARC BIB Create	PTF - Create 2	2 hr 37 min	3 hours 30 min	3 hours 27 min	Completed
6	10k MARC BIB Update	PTF - Updates Success - 6	7 min 10 s	5 min 59 sec	10 min 27 sec	Completed
7	25k MARC BIB Update	PTF - Updates Success - 6	19 min 3 s	19 min 52 sec	23 min 16 sec	Completed
8	50k MARC BIB Update	PTF - Updates Success - 6	38 min 53 sec	37 min 53 sec	40 min 52 sec	Completed
9	100k MARC BIB Update	PTF - Updates Success - 6	1 hr 23 min	1 hrs 14 min	1 hrs 2 min	Completed
10	500k MARC BIB Update	PTF - Updates Success - 6		6 hrs 39 min	5 hrs 31 min	Completed

Memory Utilization

Memory usage for both sets of tests (Creates and Updates) showing stable trend. Memory of all modules returned to normal state after tests finished. No memory leak suspects observed.

Memory usage for set of MARC BIB creates.

Memory usage for set of MARC BIB updates

CPU Utilization

CPU Usage is stable for all modules involved for creates and updates. Most used module is mod-inventory (10%).

RDS Metrics

Database CPU was stable for 10 K 25K 50K 100 K and 500K records. As expected DB CPU reached 100% and stayed same during all tests.

DB metrics for create tests

Note: on 10K and 25K Create tests mod-orders was triggered. It was disabled on 50,100,500K tests. Results was not affected.

Note: here visible that mod-search queries is not included in top 10 queries, meaning that mod-search runtime indexing issue are fixed.

DB metrics for update tests

Open Search service

Open search CPU utilisation didn’t exceeds 30% on both data and master nodes, showing stable trend.

CPU usage data nodes

CPU usage master node

MSK service

MSK service showed stable trend. Max CPU usage during tests was ±60% on one of brokers.

Disk usage on all brokers didn’t exceed 10%. (300GB of memory is allocated per broker).

Additional information from module and database logs

Deadlocks observed on DB side during creates and updates data import. These deadlock is not affecting functionality of DI itself and runtime indexing functionality too as mod-search handling deadlocks on back-end side. (Deadlocks happening during runtime reindexing when mod-search working with DB). Ticket created https://folio-org.atlassian.net/browse/MSEARCH-932

Errors

No critical errors observed during data import (Creates and Updates).

The only issue observed during 500K create import. 8 records failed to create due to data in file corruption.

Appendix

Infrastructure

PTF -environment rcon

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r6.xlarge database instances, writer
MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x (KRaft mode)
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Cluster Resources - rcon-pvt

Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 8 mod-remote-storage:3.3.3 2 4920 4472 0 3960 512 512 mod-finance-storage 7 mod-finance-storage:8.7.3 2 1024 896 0 700 88 128 mod-ncip 7 mod-ncip:1.15.6 2 1024 896 0 768 88 128 mod-agreements 8 mod-agreements:7.1.4 2 1592 1488 0 0 0 0 mod-ebsconet 8 mod-ebsconet:2.3.1 2 1248 1024 0 700 128 256 mod-organizations 7 mod-organizations:2.0.0 2 1024 896 0 700 88 128 mod-consortia 10 mod-consortia:1.2.2 2 5136 4776 0 2048 512 1024 edge-sip2 7 edge-sip2:3.3.1 2 1024 896 0 768 88 128 mod-serials-management 8 mod-serials-management:1.1.2 2 2480 2312 0 1792 384 512 mod-settings 7 mod-settings:1.1.0 2 1024 896 0 768 88 128 mod-data-import 10 mod-data-import:3.2.4 1 2048 1844 0 1292 384 512 mod-search 18 mod-search:4.0.7 2 2592 2480 0 1440 512 1024 edge-dematic 7 edge-dematic:2.3.1 1 1024 896 0 768 88 128 mod-inn-reach 4 mod-inn-reach:3.2.1-SNAPSHOT.102 2 3600 3240 0 2880 512 1024 mod-record-specifications 7 mod-record-specifications:1.0.2 2 1024 896 0 768 88 128 mod-tags 7 mod-tags:2.3.0 2 1024 896 0 768 88 128 mod-authtoken 9 mod-authtoken:2.16.1 2 1440 1152 0 922 88 128 edge-courses 8 edge-courses:1.5.1 2 1024 896 0 768 88 128 mod-notify 7 mod-notify:3.3.0 2 1024 896 0 768 88 128 mod-inventory-update 7 mod-inventory-update:4.0.0 2 1024 896 0 768 88 128 mod-configuration 7 mod-configuration:5.11.0 2 1024 896 0 768 88 128 mod-orders-storage 7 mod-orders-storage:13.8.3 2 1024 896 0 700 88 128 edge-caiasoft 7 edge-caiasoft:2.3.2 2 1024 896 0 768 88 128 mod-login-saml 7 mod-login-saml:2.9.3 2 1024 896 0 768 88 128 mod-erm-usage-harvester 7 mod-erm-usage-harvester:5.0.1 2 1024 896 0 768 88 128 mod-gobi 7 mod-gobi:2.9.0 2 1024 896 0 700 88 128 mod-licenses 7 mod-licenses:6.1.2 2 2480 2312 0 1792 384 512 mod-password-validator 7 mod-password-validator:3.3.0 2 1440 1298 0 768 384 512 edge-dcb 8 edge-dcb:1.2.1 2 1024 896 0 768 88 128 mod-bulk-operations 8 mod-bulk-operations:2.1.8 2 3072 2600 0 1536 384 512 mod-fqm-manager 10 mod-fqm-manager:3.0.7 2 3000 2600 0 768 88 128 mod-graphql 9 mod-graphql:1.13.1 2 1024 896 0 768 88 128 mod-finance 8 mod-finance:5.0.1 2 1024 896 0 700 88 128 mod-erm-usage 7 mod-erm-usage:5.0.0 2 2800 2550 0 1800 384 512 mod-batch-print 7 mod-batch-print:1.2.0 2 1024 896 0 768 88 128 mod-tlr 4 mod-tlr:1.0.0-SNAPSHOT.8 2 1024 896 0 768 88 128 mod-lists 12 mod-lists:3.0.5 2 6000 2600 0 768 88 128 mod-copycat 7 mod-copycat:1.7.0 2 1024 512 0 768 88 128 mod-entities-links 11 mod-entities-links:3.1.3 2 2592 2480 0 1440 0 1024 mod-permissions 13 mod-permissions:6.6.1 2 1684 1544 512 1024 384 512 pub-edge 7 pub-edge:2023.06.14 2 1024 896 0 768 0 0 mod-orders 9 mod-orders:12.9.9 2 2048 1740 0 1024 384 512 edge-patron 8 edge-patron:5.2.1 2 1024 896 0 768 88 128 mod-marc-migrations 26 mod-marc-migrations:1.0.0 2 1024 896 0 768 88 128 edge-ncip 8 edge-ncip:1.10.1 2 1024 896 0 768 88 128 edge-inn-reach 5 edge-inn-reach:3.3.0-SNAPSHOT.69 2 1024 896 0 768 88 128 mod-users-bl 7 mod-users-bl:7.9.3 2 1440 1152 0 922 88 128 mod-oa 4 mod-oa:2.1.0-SNAPSHOT.66 2 1024 896 0 768 88 128 mod-inventory-storage 12 mod-inventory-storage:28.0.4 2 4096 3690 0 3076 512 1024 mod-invoice 8 mod-invoice:5.9.2 2 1440 1152 0 922 88 128 mod-user-import 7 mod-user-import:3.9.0 2 1024 896 0 768 88 128 mod-sender 7 mod-sender:1.13.0 2 1024 896 0 768 88 128 edge-oai-pmh 7 edge-oai-pmh:2.10.0 2 1512 1360 0 1440 384 512 mod-data-export-worker 10 mod-data-export-worker:3.3.6 2 3072 2048 0 2048 384 512 mod-rtac 7 mod-rtac:3.7.0 2 1024 896 0 768 88 128 mod-circulation-storage 8 mod-circulation-storage:17.3.3 2 2880 2592 0 1814 384 512 mod-source-record-storage 13 mod-source-record-storage:5.9.5 2 5600 5000 0 3500 384 512 mod-calendar 7 mod-calendar:3.2.0 2 1024 896 0 768 88 128 mod-event-config 7 mod-event-config:2.8.0 2 1024 896 0 768 88 128 mod-courses 8 mod-courses:1.4.11 2 1024 896 0 768 88 128 mod-circulation-item 7 mod-circulation-item:1.1.0 2 1024 896 0 0 0 0 mod-inventory 9 mod-inventory:21.0.5 2 2880 2592 0 1814 384 512 mod-email 8 mod-email:1.18.1 2 2800 2550 0 1800 384 512 mod-requests-mediated 4 mod-requests-mediated:1.0.0-SNAPSHOT.4 2 1024 896 0 768 88 128 mod-circulation 8 mod-circulation:24.3.8 2 2880 2592 0 1814 384 512 mod-pubsub 8 mod-pubsub:2.15.3 2 1536 1440 0 922 384 512 mod-di-converter-storage 9 mod-di-converter-storage:2.3.1 2 1024 896 0 768 88 128 edge-rtac 7 edge-rtac:2.8.0 2 1024 896 0 768 88 128 edge-orders 7 edge-orders:3.1.0 2 1024 896 0 768 88 128 mod-users 8 mod-users:19.4.5 2 1024 896 0 768 88 128 mod-template-engine 7 mod-template-engine:1.21.0 2 1024 896 0 768 88 128 mod-patron-blocks 7 mod-patron-blocks:1.11.1 2 1024 896 0 768 88 128 mod-audit 8 mod-audit:2.10.2 2 1024 896 0 768 88 128 edge-fqm 9 edge-fqm:3.0.2 2 1024 896 0 768 88 128 mod-source-record-manager 8 mod-source-record-manager:3.9.5 2 5600 5000 0 3500 384 512 nginx-edge 7 nginx-edge:2023.06.14 2 1024 896 0 0 0 0 mod-quick-marc 7 mod-quick-marc:6.0.0 1 2288 2176 0 1664 384 512 nginx-okapi 7 nginx-okapi:2023.06.14 2 1024 896 0 0 0 0 okapi-b 8 okapi:6.1.1 3 1684 1440 1024 922 384 512 mod-feesfines 7 mod-feesfines:19.2.1 2 1024 896 0 768 88 128 mod-invoice-storage 7 mod-invoice-storage:5.9.1 2 1872 1536 0 1024 384 512 mod-reading-room 7 mod-reading-room:1.0.0 2 1024 896 0 768 88 128 mod-dcb 8 mod-dcb:1.2.4 2 1024 896 0 768 88 128 mod-service-interaction 7 mod-service-interaction:4.1.1 2 2048 1844 0 1290 384 512 mod-patron 8 mod-patron:6.2.5 2 1024 896 0 768 88 128 mod-data-export 13 mod-data-export:5.1.5 1 2048 1844 0 0 0 0 mod-oai-pmh 7 mod-oai-pmh:3.14.3 2 4096 3690 0 3076 384 512 edge-connexion 7 edge-connexion:1.3.1 2 1024 896 0 768 88 128 mod-notes 7 mod-notes:6.0.0 2 1024 896 0 952 384 512 mod-kb-ebsco-java 8 mod-kb-ebsco-java:5.0.0 2 1024 896 0 768 88 128 mod-organizations-storage 7 mod-organizations-storage:4.8.1 2 1024 896 0 700 88 128 mod-data-export-spring 8 mod-data-export-spring:3.4.3 1 2048 1844 0 1536 384 512 mod-login 7 mod-login:7.12.1 2 1440 1298 0 768 384 512 pub-okapi 7 pub-okapi:2023.06.14 2 1024 896 0 768 0 0 edge-erm 5 edge-erm:1.3.0 2 1024 896 0 768 88 128 mod-eusage-reports 7 mod-eusage-reports:3.0.0 2 1024 896 0 768 88 128

R - Ramsons release

Dataset Size:

Record type	Number of records

Record type	Number of records
Instances	1 163 924
Holdings	1 348 036
Items	2 091 901

Methodology/Approach

Pre-generated files were used for DI Create job profile
- 10K, 25K, 50K, 100K and 500K files.
Run DI Create on a single tenant(cs00000int_0001) one by one with the delay with files using PTF - Create 2 profile.
Prepare files for DI Update with the Data export app, using previously imported items
Run DI Update on a single tenant(cs00000int_0001) one by one with the delay with prepared files using PTF - Update Success 2 profile

1K, 10K, 25K, 50K, 100K and 500K files.

Data-import durations were obtained from DB using SQL query

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from [tenant]_mod_source_record_manager.job_execution order by started_date desc limit 1000;

Data Import test report Ramsons [ECS]