Overview

...

This document contains the results of testing Data Export (MARC BIB) on Quesnelia release with Data Export tests for 1k, 100k, 500k. Three csv files were prepared to run Data Export with Default instances export job profile and srs - holdings and items job profiles.
...
DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.

Recommendations & Jiras

Consider increasing of cpu for mod-data-export module to smoothen spikes with big size files.

Test Results

This table contains durations for jobs with 3 job profiles.

...

Profile	CSV File	DE Duration/Status Poppy 2 set		DE Duration/Status Quesnelia		DE Duration, DELTA Poppy/Quesnelia
		Result	Status	Result	Status	hh:mm:ss
DE MARC Bib (Default instances export job profile)	1kDE.csv	00:00:08	COMPLETED	00:00:02	COMPLETED	00:00:06 - 4 times improvement
	100kDE.csv	00:15:36	COMPLETED	00:02:17	COMPLETED	00:13:19 - 7 times improvement
	500kDE.csv	00:57:25	FAIL	00:05:10	COMPLETED
DE MARC Bib (srs - holdings and items)	1kDE.csv	00:00:29	COMPLETED	00:00:04	COMPLETED	00:00:25 - 7 times improvement
	100kDE.csv	00:47:23	COMPLETED	00:05:13	COMPLETED	00:42:10 - 9 times improvement
	500kDE.csv	04:11:09	FAIL	00:08:58	COMPLETED

Instance CPU Utilization

Service CPU Utilization

Expand

title	CPU utilization

Default instances export job profile with 500k file

Module	CPU
mod-data-export-b	434.91
mod-inventory-b	10.79
mod-source-record-manager-b	1.74
mod-source-record-storage-b	1.5
okapi-b	0.96
mod-users-bl-b	0.63
mod-authtoken-b	0.61
mod-inventory-storage-b	0.39
nginx-okapi	0.24
pub-okapi	0.16

srs - holdings and items with 500k file

Module	CPU
mod-data-export-b	296.31
mod-inventory-b	12.79
mod-source-record-manager-b	1.85
mod-source-record-storage-b	1.49
okapi-b	1.1
mod-authtoken-b	0.95
mod-users-bl-b	0.69
mod-inventory-storage-b	0.54
nginx-okapi	0.32
pub-okapi	0.21

...

TOP 20 modules

Module	CPU
mod-data-export-b

For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.

For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.

Image Removed

For srs - holdings and items job profile - 100k - 63%, 500k - 296%.

Image Removed

Memory Utilization

Expand

title	Memory consumption

Module	Memory
mod-data-export-b	97
mod-inventory-b	55
okapi-b	42
mod-source-record-manager-b	42
mod-users-bl-b	32
mod-source-record-storage-b	27
mod-authtoken-b	25
mod-inventory-storage-b	14
nginx-okapi	5
pub-okapi	4296.31
mod-consortia-b	19.03
mod-inventory-b	12.79
mod-dcb-b	8.17
mod-quick-marc-b	7.99
mod-pubsub-b	7.05
mod-users-b	6.85
mod-audit-b	6.18
mod-kb-ebsco-java-b	5.78
edge-dematic-b	4.5
mod-erm-usage-harvester-b	4.49
mod-organizations-storage-b	4.04
mod-licenses-b	3.51
mod-permissions-b	3.06
mod-data-export-spring-b	2.78
mod-lists-b	2.77
mod-organizations-b	2.74
mod-user-import-b	2.7
mod-tags-b	2.57
mod-sender-b	2.5

Mod-data-export-b

For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.

For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.

Image Added

For srs - holdings and items job profile - 100k - 63%, 500k - 296%.

Image Added

Memory Utilization

Expand

title	Memory consumption

Module	Memory
mod-data-export-b	97
mod-inventory-b	55
okapi-b	42
mod-source-record-manager-b	42
mod-users-bl-b	32
mod-source-record-storage-b	27
mod-authtoken-b	25
mod-inventory-storage-b	14
nginx-okapi	5
pub-okapi	4

TOP 20 modules srs - holdings and items with 500k file

Module	Memory
mod-data-export-b	97.51
mod-data-export-worker-b	89.71
mod-dcb-b	78.72
mod-consortia-b	78.13
mod-oa-b	76.73
mod-orders-b	65.1
mod-copycat-b	62.06
mod-calendar-b	56.01
mod-agreements-b	55.19
mod-invoice-b	55.1
mod-permissions-b	55.07
mod-circulation-item-b	53.84
mod-erm-usage-harvester-b	52.36
mod-service-interaction-b	51.65
mod-notes-b	50.5
mod-orders-storage-b	50.49
mod-users-b	49.88
mod-tags-b	47.66
mod-inventory-b	45.58
mod-audit-b	44.97

This graph contains DE related modules.

...

Default instances export job profile

autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING srs - holdings and items job profile

autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids

select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)

select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

Errors / Additional information

...

3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;

All tests were carried out sequentially with each job profile on main tenant fs09000000.

To get status and time range for export jobs the query used:

Expandcode

language	sql
theme	FadeToGrey
title	QueriesSQL Query

select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'srs - holdings and items'


ORDER BY completed_date DESC



select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'Default instances export job profile'


ORDER BY completed_date DESC



select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'Export for Data Import updates(test)1'


ORDER BY completed_date DESC

Infrastructure

PTF -environment qcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
Data set for fs09000000
- Instances - 25606331
- Items - 26779913
- Holdings - 25576735
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

title

Expand

title	All qcp1 modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt
Tue Jun 04 07:31:53 UTC 2024
mod-remote-storage	4	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-ncip	4	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	4	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-agreements	4	mod-agreements:7.0.0	2	1592	1488	128	0	0	0
mod-ebsconet	4	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-organizations	4	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-consortia	2	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
edge-sip2	2	edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	128	768	88	128
mod-serials-management	4	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512
mod-settings	4	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-data-import	7	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
edge-dematic	4	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-search	4	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024
mod-inn-reach	2	mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	1024	2880	512	1024
mod-tags	4	mod-tags:2.2.0	2	1024	896	128	768	88	128
edge-courses	4	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-authtoken	5	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
mod-inventory-update	4	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	4	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	4	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	4	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	4	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	4	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	4	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-licenses	4	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	4	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-password-validator	4	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-bulk-operations	4	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	4	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	4	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	5	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	4	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	4	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	5	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	4	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	4	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	5	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	8	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
pub-edge	3	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	4	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	4	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	4	edge-ncip:1.9.2	2	1024	896	128	768	88	128
edge-inn-reach	2	edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	128	768	88	128
mod-users-bl	4	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	128	768	88	128
mod-inventory-storage	4	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-invoice	5	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-user-import	4	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	5	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	4	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	4	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	4	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	4	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-calendar	4	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-source-record-storage	4	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-event-config	4	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	4	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	4	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	4	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	4	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-pubsub	4	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-circulation	4	mod-circulation:24.2.0	2	2880	2592	1536	1814	384	512
mod-di-converter-storage	4	mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128
edge-rtac	4	edge-rtac:2.7.1	2	1024	896	128	768	88	128
edge-orders	4	edge-orders:3.0.0	2	1024	896	128	768	88	128
mod-users	5	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-template-engine	4	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	4	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	4	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	4	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	5	mod-source-record-manager:3.9.0-SNAPSHOT.330	2	5600	5000	2048	3500	384	512
nginx-edge	3	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	4	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	3	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	4	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	4	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	4	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-dcb	5	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-service-interaction	4	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-data-export	11	mod-data-export:5.0.4	1	2048	1524	1024	0	0	0
mod-patron	4	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-oai-pmh	4	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	4	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java	4	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	4	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-data-export-spring	4	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
mod-organizations-storage	4	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-login	4	mod-login:7.11.0	2	1440	1298	1024	768	384	512
pub-okapi	3	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	4	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Expand

Data Export related modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt
Tue Jun 04 07:31:53 UTC 2024
mod-authtoken	5	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
mod-users-bl	4	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-source-record-storage	4	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-inventory	4	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-source-record-manager	5	mod-source-record-manager:3.9.0-SNAPSHOT.330	2	5600	5000	2048	3500	384	512
nginx-okapi	3	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	4	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-data-export	11	mod-data-export:5.0.4	1	2048	1524	1024	0	0	0
pub-okapi	3	pub-okapi:2023.06.14	2	1024	896	128	768	0	0

Versions Compared

Old Version 2

New Version Current

Key

Table of Contents

Overview

Recommendations & Jiras

Test Results

Instance CPU Utilization

Service CPU Utilization

Memory Utilization

Memory Utilization

Errors / Additional information

Infrastructure

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Page Comparison

Versions Compared

Old Version 2

New Version Current

Key

Overview

Recommendations & Jiras

Test Results

Instance CPU Utilization

Service CPU Utilization

Memory Utilization

Memory Utilization

Errors / Additional information

Infrastructure