1 Overview
2 Summary
3 Results
- 3.1 Memory Utilization
- 3.2 CPU Utilization
- 3.3 RDS Metrics
- 3.4 MSK service
4 Appendix
- 4.1 Infrastructure
- 4.2 Cluster Resources - secon-pvt
- 4.3 Methodology/Approach

Overview

This document contains the results of testing Data Import for MARC Bibliographic records creates and updates with different file sizes (10K, 25K, 50K, 100K, 500K) at Sunflower release [ECS].

Note: Starting from Sunflower all releases are Eureka.
Ticket: https://folio-org.atlassian.net/browse/PERF-1178

Summary

All Data-imports jobs finished successfully without errors.
Duration of data imports for creates and updates are mostly the same as was in Sunflower GA
DI duration growth correlates to the number of records imported.
No memory leak is suspected for DI modules.
Services CPU utilization, Service memory utilization, and DB CPU utilization have the same utilization trend and values as in the R release.
During testing, we discovered a slight dependency on mod-orders. With the module disabled, we see better performance. A ticket has been created for Thunderjet team for a more detailed analysis of the issue: https://folio-org.atlassian.net/browse/MODORDSTOR-483

Results

Test #	Data-import test	Profile	Duration Sunflower CSP1 enabled orders	Duration Sunflower CSP1 disabled orders	Duration Sunflower enabled orders	Duration Sunflower disabled orders-storage	Results

Test #	Data-import test	Profile	Duration Sunflower CSP1 enabled orders	Duration Sunflower CSP1 disabled orders	Duration Sunflower enabled orders	Duration Sunflower disabled orders-storage	Results
1	5k MARC BIB Create	PTF - Create 2	3 min 3 s	-	2 min 32 s	1 min 26 s	Completed
2	10k MARC BIB Create	PTF - Create 2	5 min 32 s	-	4 min 46 s	2 min 50 s	Completed
3	25k MARC BIB Create	PTF - Create 2	12 min 25 s	-	11 min 3 s	6 min 48 s	Completed
4	50k MARC BIB Create	PTF - Create 2	26 min 20 s	18 min 05 s	20 min 44 s	13 min 37 s	Completed
5	100k MARC BIB Create	PTF - Create 2	54 min 8 s	39 min 51 s	46 min 24 s	30 min 3 s	Completed
6	5k MARC BIB Update	PTF - Updates Success - 6	2 min 44 s	-	-	3 min 53 s	Completed
7	10k MARC BIB Update	PTF - Updates Success - 6	5 min 33 s	-	-	7 min 16s	Completed
8	25k MARC BIB Update	PTF - Updates Success - 6	12 min 51 s	-	-	18 min 38 s	Completed
9	50k MARC BIB Update	PTF - Updates Success - 6	25 min 32 s	-	-	37 min 55 s	Completed
10	100k MARC BIB Update	PTF - Updates Success - 6	54 min 41 s	-	-	1 hr 22 min	Completed

Memory Utilization

Memory utilisation showing stable trend, without spikes and drops. No signs

CPU Utilization

RDS Metrics

MSK service

Appendix

Infrastructure

PTF -environment secon

8 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r7g.xlarge database instances, writer
MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x (KRaft mode)
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Cluster Resources - secon-pvt

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size	R/W Split Enabled

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size	R/W Split Enabled
mod-remote-storage	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.4.3	2	4920	4472	0	3960	512	512	false
mod-remote-storage - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-finance-storage	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.8.4	2	1024	896	0	700	88	128	false
mod-finance-storage - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-ncip	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.15.7	2	1024	896	0	768	88	128	false
mod-ncip - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-agreements	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.2.2	2	4096	4096	0	4096	0	0	false
mod-agreements - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-ebsconet	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.4.0	2	1248	1024	0	700	128	256	false
mod-ebsconet - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-organizations	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:2.1.0	2	1024	896	0	620	88	128	false
mod-organizations - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-serials-management	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:2.0.3	2	2780	2312	0	1792	384	896	false
mod-serials-management - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-settings	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.2.0	2	1024	896	0	768	88	128	false
mod-settings - Sidecar 1	N/A	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/folio-module-sidecar:3.0.8	N/A	1024	512	0	256	0	96	false
mod-data-import	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.3.5

Data Import BIB Sunflower CSP1 [ECS]