Overview

This document contains the results of testing Data Export (MARC BIB) on the Ramsons [NON-ECS] release on rcp1 environment.

PERF-968 - Getting issue details... STATUS

Summary

Data Export tests finished successfully on rcp1 environment using the profiles Default instances export job profile and SRS - holdings and items job profile.
Comparing with previous testing results Quesnelia and Ramsons releases
- Data Export processed all files including file with 500k records without errors for Ramsons releases.
- Data Export durations improved - around 10% in Average for Ramsons releases.

Test Results

This table contains durations for Data Export with two job profiles.

Profile	CSV File	Tenant (fs09000000)
Profile	CSV File	Result	Status
DE MARC Bib (Default instances export job profile)	1k.csv	0:00:03	COMPLETED
	100k.csv	0:02:19	COMPLETED
	500k.csv	0:04:33	COMPLETED
DE MARC Bib (SRS - holdings and items job profile)	1k.csv	0:00:06	COMPLETED
	100k.csv	0:07:02	COMPLETED
	500k.csv	0:08:57	COMPLETED

Comparison

This table contains durations comparison between Ramsons and Quesnelia releases.

Profile	CSV File	DE Duration Ramsons	DE Duration Quesnelia	DE Duration, DELTA Poppy/Quesnelia
Profile	CSV File	Result hh:mm:ss	Result hh:mm:ss	hh:mm:ss / percent
DE MARC Bib (Default instances export job profile)	1k.csv	0:00:03	0:00:02	+00:00:01
	100k.csv	0:02:19	0:02:39	-00:00:20 -12%
	500k.csv	0:04:33	0:05:21	-00:00:48 -15%
DE MARC Bib (SRS - holdings and items job profile)	1k.csv	0:00:06	0:00:05	+00:00:01
	100k.csv	0:07:02	0:08:15	-00:01:13 -15%
	500k.csv	0:08:57	0:09:22	-00:00:25 -5%

Resource utilization for Test #1 and Test #2

Resource utilization table

CPU		RAM
mod-data-export-b	42%	mod-data-export-b	79%
mod-inventory-b	1.40%	mod-inventory-b	64%
mod-source-record-storage-b	0.40%	okapi-b	51%
mod-source-record-manager-b	0.10%	mod-users-bl-b	42%
mod-inventory-storage-b	0.10%	mod-source-record-storage-b	28%
okapi-b	0.10%	mod-inventory-storage-b	27%
mod-authtoken-b	0.10%	mod-authtoken-b	18%
mod-users-bl-b	0.10%	mod-source-record-manager-b	18%
nginx-okapi	0.10%	nginx-okapi	4%

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-data-export used 42% CPU in spike.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU spike was 45%.

DB Connections

DB connections was 1134.

Kafka metrics

OpenSearch Data Nodes metrics

DB load

Top SQL-queries

Top applications

Appendix

Infrastructure

PTF - environment Ramsons NON-ECS (rcp1)

rcp1 6 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge database instance: Writer instance
MSK fse-test
- - 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
  - Apache Kafka version 3.7.x, metadata mode - KRaft
  - EBS storage volume per broker 300 GiB
  - auto.create.topics.enable=true
  - log.retention.minutes=480
  - default.replication.factor=2
  - revision - 26
OpenSearch 2.13 ptf-test cluster
- r6g.2xlarge.search 4 data nodes
- r6g.large.search 3 dedicated master nodes

Ramsons modules memory and CPU parameters

Cluster Resources - rcp1-pvt (Thu Jan 30 12:56:18 UTC 2025)

Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
12	mod-remote-storage:3.3.5	2	4920	4472	0	3960	512	512
9	mod-ncip:1.15.6	2	1024	896	0	768	88	128
10	mod-finance-storage:8.7.3	2	1024	896	0	700	88	128
12	mod-agreements:7.1.5	2	1592	1488	0	0	0	0
11	mod-ebsconet:2.3.1	2	1248	1024	0	700	128	256
9	mod-organizations:2.0.0	2	1024	896	0	700	88	128
4	mod-consortia:1.2.2	2	5136	4776	0	4416	512	1024
10	edge-sip2:3.3.1	2	1024	896	0	768	88	128
10	mod-settings:1.1.0	2	1024	896	0	768	88	128
12	mod-serials-management:1.1.3	2	2480	2312	0	1792	384	512
9	edge-dematic:2.3.1	1	1024	896	0	768	88	128
10	mod-data-import:3.2.4	1	2048	1844	0	1292	384	512
24	mod-search:4.0.8	2	2592	2480	0	1440	512	1024
5	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024
18	mod-record-specifications:1.0.2	2	1024	896	0	768	88	128
9	mod-tags:2.3.0	2	1024	896	0	768	88	128
10	mod-authtoken:2.16.1	2	1440	1152	0	922	88	128
11	edge-courses:1.5.1	2	1024	896	0	768	88	128
9	mod-notify:3.3.0	2	1024	896	0	768	88	128
9	mod-inventory-update:4.0.0	2	1024	896	0	768	88	128
9	mod-configuration:5.11.0	2	1024	896	0	768	88	128
10	mod-orders-storage:13.8.3	2	1024	896	0	700	88	128
9	edge-caiasoft:2.3.2	2	1024	896	0	768	88	128
9	mod-login-saml:2.9.3	2	1024	896	0	768	88	128
10	mod-erm-usage-harvester:5.0.2	2	1024	896	0	768	88	128
9	mod-password-validator:3.3.0	2	1440	1298	0	768	384	512
9	mod-gobi:2.9.0	2	1024	896	0	700	88	128
10	mod-licenses:6.1.2	2	2480	2312	0	1792	384	512
10	edge-dcb:1.2.2	2	1024	896	0	768	88	128
12	mod-bulk-operations:2.1.10	2	3072	2600	0	1536	384	512
24	mod-fqm-manager:3.0.10	2	3000	2600	0	2048	384	512
11	mod-graphql:1.13.1	2	1024	896	0	768	88	128
11	mod-finance:5.0.1	2	1024	896	0	700	88	128
9	mod-erm-usage:5.0.0	2	1024	896	0	768	88	128
9	mod-batch-print:1.2.0	2	1024	896	0	768	88	128
5	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
19	mod-lists:3.0.8	2	6000	2600	0	768	88	128
9	mod-copycat:1.7.0	2	1024	512	0	768	88	128
15	mod-entities-links:3.1.3	2	2592	2480	0	1440	0	1024
20	mod-permissions:6.6.1	2	1684	1544	0	1024	384	512
7	pub-edge:2023.06.14	2	1024	896	0	768	0	0
12	mod-orders:12.9.10	2	2048	1440	0	1024	384	512
11	edge-patron:5.2.1	2	1024	896	0	768	88	128
9	edge-ncip:1.10.1	2	1024	896	0	768	88	128
31	mod-marc-migrations:1.0.0	2	1024	896	0	768	88	128
4	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
10	mod-users-bl:7.9.3	2	1440	1152	0	922	88	128
6	mod-oa:2.1.0-SNAPSHOT.66	2	2048	896	0	1800	88	512
11	mod-invoice:5.9.2	2	1440	1152	0	922	88	128
18	mod-inventory-storage:28.1.0-SNAPSHOT.792	2	4096	3690	0	3076	512	1024
9	mod-user-import:3.9.0	2	1024	896	0	768	88	128
9	mod-sender:1.13.0	2	1024	896	0	768	88	128
10	edge-oai-pmh:2.10.0	2	1512	1360	0	1440	384	512
12	mod-data-export-worker:3.3.8	2	3072	2048	0	2048	384	512
9	mod-rtac:3.7.0	2	1024	896	0	768	88	128
12	mod-circulation-storage:17.3.5	2	2880	2592	0	1814	384	512
17	mod-source-record-storage:5.9.7	2	5600	5000	0	3500	384	512
10	mod-calendar:3.2.0	2	1024	896	0	768	88	128
9	mod-event-config:2.8.0	2	1024	896	0	768	88	128
9	mod-courses:1.4.11	2	1024	896	0	768	88	128
9	mod-circulation-item:1.1.0	2	1024	896	0	0	0	0
12	mod-inventory:21.0.9	2	2880	2592	0	1814	384	512
11	mod-email:1.18.1	2	1024	896	0	768	88	128
13	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
10	mod-di-converter-storage:2.3.1	2	1024	896	0	768	88	128
12	mod-pubsub:2.15.4	2	1536	1440	0	922	384	512
12	mod-circulation:24.3.11	2	2880	2592	0	1814	384	512
9	edge-orders:3.1.0	2	1024	896	0	768	88	128
9	edge-rtac:2.8.0	2	1024	896	0	768	88	128
9	mod-template-engine:1.21.0	2	1024	896	0	768	88	128
11	mod-users:19.4.5	2	1024	896	0	768	88	128
10	mod-patron-blocks:1.11.1	2	1024	896	0	768	88	128
11	mod-audit:2.10.2	2	1024	896	0	768	88	128
11	edge-fqm:3.0.2	2	1024	896	0	768	88	128
16	mod-source-record-manager:3.9.7	2	5600	5000	0	3500	384	512
7	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
9	mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512
7	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
10	okapi:6.1.1	3	2024	1440	0	1024	384	512
10	mod-feesfines:19.2.2	2	1024	896	0	768	88	128
10	mod-invoice-storage:5.9.1	2	1872	1536	0	1024	384	512
10	mod-reading-room:1.0.0	2	1024	896	0	768	88	128
11	mod-service-interaction:4.1.1	2	2048	1844	0	1290	384	512
12	mod-dcb:1.2.6	2	1024	896	0	768	88	128
14	mod-data-export:5.1.6	1	2048	1844	0	0	0	0
12	mod-patron:6.2.6	2	1024	896	0	768	88	128
11	mod-oai-pmh:3.14.4	2	4096	3690	0	3076	384	512
9	edge-connexion:1.3.1	2	1024	896	0	768	88	128
9	mod-notes:6.0.0	2	1024	896	0	952	384	512
9	mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128
10	mod-login:7.12.1	2	1440	1298	0	768	384	512
9	mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128
11	mod-data-export-spring:3.4.3	1	2048	1844	0	1536	384	512
7	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
8	edge-erm:1.3.0	2	1024	896	0	768	88	128
9	mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128

Methodology/Approach

Data Export tests scenario using the profiles Default instances export job profile and SRS - holdings and items were started from UI on Ramsons (rcp1) NON-ECS environment.

Test set

Test 1: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant (fs09000000) only using Default instances export job profile.
Test 2: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant (fs09000000) only using SRS - holdings and items job profile.

To get status and time range for export jobs the query used:

SQL Query

SELECT 
jsonb->>'status' AS status,
to_timestamp((jsonb->>'startedDate')::bigint / 1000) AS startedDate,
to_timestamp((jsonb->>'completedDate')::bigint / 1000) AS completedDate,
job_profile_name,
exported_file->>'fileName' AS fileName,

to_char(
(to_timestamp((jsonb->>'completedDate')::bigint / 1000) - to_timestamp((jsonb->>'startedDate')::bigint / 1000))::interval, 
'HH24:MI:SS'
) AS duration_hhmmss,
(jsonb->>'completedDate')::bigint - (jsonb->>'startedDate')::bigint AS duration_ms
FROM 
fs09000000_mod_data_export.job_executions,
jsonb_array_elements(jsonb->'exportedFiles') AS exported_file
WHERE 
(jsonb->>'hrId')::int IN (11128, 11129, 11130, 11131, 11132, 11133)
ORDER BY 
jsonb->>'startedDate' ASC
LIMIT 10;

Folio Development Teams

PTF - Data Export Test Report (Ramsons) [NON-ECS]