Table of Contents

outline	true

Overview

This document contains the results of testing Data Export (MARC BIB) on the Quesnelia [ECS] release on qcon environment.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-844

Summary

Data import tests finished successfully, only Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files. Duration of DI grew in correspondence with the number of records in files.
Check-in and Check-out with 5 virtual users was performed during DI Create new MARC authority records jobs for non-matches No issues.
Data Import in Quesnelia without CICO perform faster than with it.
Comparing Poppy and Quesnelia releases
- Check-in / Check-out perform better in Quesnelia. Response time improved during Create jobs for long period of work time on 15% in Average.
- DI durations improved - 11%-14% in Average.
During testing, we noticed spikes in the mod permissions module. To mitigate this issue and prevent system slowdowns, we adjusted the order of loading files, starting with Tenant 3 (qcp1-02), followed by Tenant 2 (qcp1-01), and finally Tenant 1 (qcp1-00).

Test Results

This table contains durations for jobs with 2 job profiles.

...

Profile

...

Comparison

This table contains durations comparison between Poppy and Quesnelia releases.

Resource utilization for Test #1 and Test #2

...

title	Resource utilization table

...

Service CPU Utilization

Here we can see that mod-data-export used 452% CPU in spike.

Image Removed

Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Removed

DB CPU Utilization

DB CPU spike was 32%.

Image Removed

DB Connections

DB connections was 1470.

Image Removed

DB load

Image Removed

Top SQL-queries

Image Removed

...

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

...

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

...

Table of Contents

outline	true

Overview

This document contains the results of testing Data Export (MARC BIB) on the Quesnelia [ECS] release on qcon environment.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-844

Summary

Data Export tests finished successfully on qcon environment using the profiles Default instances export job profile and srs - holdings and items job profile.
Data Export test were run on College and Central tenants, but results for comparing between environment releases were taken from College tenant.
Comparing with previous testing results Poppy and Quesnelia releases
- Data Export processed all files including file with 500k records without errors for Quesnelia releases.
- Data Export durations improved - 80% in Average for Quesnelia releases.
During testing, we noticed spikes in the mod-data-export up to 593% CPU.
For Test №5 Data Export started on College tenant(cs00000int_0001), Central tenant(cs00000int) and Professional tenant(cs00000int_0002) concurrently using the Default instances export job profile, we observed that the CPU usage of the mod-data-export module was initially at 44% before the test began then it spiked to 109% during the test and remained elevated without returning to the initial state.

Test Results

This table contains durations for Data Export with 2 job profiles.

Profile	CSV File	Tenant College (cs00000int_0001)		Central Tenant (cs00000int)
Profile	CSV File	Result	Status	Result	Status
DE MARC Bib (Default instances export job profile)	1k.csv	0:00:02	COMPLETED	0:00:05	COMPLETED
	100k.csv	0:02:39	COMPLETED	0:04:24	COMPLETED
	500k.csv	0:05:21	COMPLETED	0:06:17	COMPLETED
DE MARC Bib (srs - holdings and items)	1k.csv	0:00:05	COMPLETED	0:00:05	COMPLETED
	100k.csv	0:08:15	COMPLETED	0:05:58	COMPLETED
	500k.csv	0:09:22	COMPLETED	0:08:28	COMPLETED

This table contains durations for Test №5 Data Export for 3 tenants concurrently.

Tenant

CSV File

Result

Status

Tenant College

(cs00000int_0001)

500k.csv

0:10:24

COMPLETED

Tenant Professional

(cs00000int_0002)

500k.csv

0:06:47

COMPLETED

Central Tenant

(cs00000int)

500k.csv

0:07:56

COMPLETED

Comparison

This table contains durations comparison between Poppy and Quesnelia releases.

Profile	CSV File	DE Duration/Status Orchid		DE Duration/Status Poppy 1 set		DE Duration/Status Quesnelia Tenant College (cs00000int_0001)		DE Duration, DELTA Poppy/Quesnelia
Profile	CSV File	Result	Status	Result	Status	Result	Status	hh:mm:ss / percent
DE MARC Bib (Default instances export job profile)	1k.csv			00:00:08	COMPLETED	0:00:02	COMPLETED	-00:00:06 -75%
	100k.csv			00:15:36	COMPLETED	0:02:39	COMPLETED	-00:12:57 -83.02%
	500k.csv			00:57:25	FAIL	0:05:21	COMPLETED	-00:52:04 -90.68%
DE MARC Bib (srs - holdings and items)	1k.csv	00:00:27	COMPLETED	00:00:29	COMPLETED	0:00:05	COMPLETED	-00:00:24 -82.76%
	100k.csv	00:47:51	COMPLETED	00:47:23	COMPLETED	0:08:15	COMPLETED	-00:39:08 -82.59%
	500k.csv	04:00:26	COMPLETED	04:11:09	FAIL	0:09:22	COMPLETED	-04:01:47 -96.27%

Resource utilization for Test #1 and Test #2

Expand

title	Resource utilization table

CPU		RAM
mod-data-export-b	452%	mod-data-export-b	75%
mod-inventory-b	13%	mod-source-record-manager-b	53%
mod-source-record-storage-b	2.40%	mod-inventory-b	48%
mod-source-record-manager-b	1.80%	okapi-b	32%
okapi-b	1.10%	mod-source-record-storage-b	30%
mod-authtoken-b	0.90%	mod-authtoken-b	20%
mod-users-bl-b	0.50%	mod-users-bl-b	19%
nginx-okapi	0.40%	mod-inventory-storage-b	16%
mod-inventory-storage-b	0.40%	nginx-okapi	5%

Service CPU Utilization

Here we can see that mod-data-export used 452% CPU in spike.

Image Added

Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Added

DB CPU Utilization

DB CPU spike was 32%.

Image Added

DB Connections

DB connections was 1470.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

#	TOP 5 SQL statements
1	`INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING`
2	`INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING`
3	`select mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$`
4	`select iwhe1_0.id,iwhe1_0.hrid from v_instance_hrid iwhe1_0 where iwhe1_0.id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$79,$80,$81,$82,$83,$84,$85,$86,$87,$88,$89,$90,$91,$92,$93,$94,$95,$96,$97,$98,$99,$100,$101,$102,$103,$104,$105,$1`
5	`select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1`

...

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 35%.

DB Connections

DB connections was 1377.

DB load

...


select mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
select eie1_0.id,eie1_0.instance_id,_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.job_executioninstance_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
1 instance of db.r6.xlarge database instance: Writer instance
OpenSearch
- domain: fse
- Number of nodes: 9
- Version: OpenSearch_2_7_R20240502
MSK - tenat
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Kafka consolidated topics enabled

...

title	Quesnelia modules memory and CPU parameters

...

 offset $4 rows fetch first $5 rows only

Resource utilization for Test #5

Expand

title	Resource utilization table

CPU		RAM
mod-data-export-b	592%	mod-data-export-b	108%
mod-inventory-b	10%	mod-inventory-b	78%
mod-source-record-storage-b	1.80%	mod-source-record-storage-b	40%
mod-authtoken-b	1.70%	mod-source-record-manager-b	39%
mod-source-record-manager-b	1.50%	okapi-b	32%
okapi-b	1.50%	mod-users-bl-b	24%
mod-inventory-storage-b	0.60%	mod-authtoken-b	18%
mod-users-bl-b	0.60%	mod-inventory-storage-b	13%
nginx-okapi	0.40%	nginx-okapi	4%

Service CPU Utilization

Here we can see that mod-data-export used 593% CPU in spike.

Image Added

Service Memory Utilization

We observed that the CPU usage of the mod-data-export module was initially at 44% before the test began. It spiked to 109% during the test and remained elevated without returning to the initial state.

Image Added

DB CPU Utilization

DB CPU was 50%.

Image Added

DB Connections

DB connections was 1368.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
1 instance of db.r6.xlarge database instance: Writer instance
OpenSearch
- domain: fse
- Number of nodes: 9
- Version: OpenSearch_2_7_R20240502
MSK - tenat
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Kafka consolidated topics enabled

Expand

title	Quesnelia modules memory and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcon-pvt
Thu May 23 10:47:25 UTC 2024
mod-remote-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreementsremote-storage:73.02.0	2	15924920	14884472	1281024	039600	512	0512
mod-ebsconetfinance-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-ebsconetstorage:28.26.0	2	12481024	896	1024	128700700	88	128256
mod-consortiancip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortiancip:1.114.04	2	30721024	2048896	128	2048768	51288	1024128
mod-organizationsagreements	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizationsagreements:17.90.0	2	10241592	8961488	128	7000	880	1280
mod-serials-managementebsconet	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-managementebsconet:12.02.0	2	24801248	23121024	128	1792700	384128	512256
mod-settingsconsortia	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settingsconsortia:1.1.0.3	2	10243072	8962048	200128	7682048	88512	1281024
mod-searchorganizations	91	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-searchorganizations:31.39.0-SNAPSHOT.224	2	25921024	2480896	2048128	1440700	51288	1024128
edgemod-serials-dematicmanagement	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-serials-dematicmanagement:21.20.01	2	10242480	8962312	128	7681792	88384	128512
mod-data-importsettings	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-importsettings:3.1.0.31	2	20481024	1844896	256200	1292768	38488	512128
mod-tagssearch	19	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tagssearch:23.23.0-SNAPSHOT.224	2	10242592	8962480	1282048	7681440	88512	1281024
modedge-authtokendematic	31	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-authtokendematic:2.152.0	1	21024	1440896	1152128	512768	922	88	128
edgemod-data-coursesimport	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-data-coursesimport:3.1.4.0	21	10242048	8961844	128256	7681292	88384	128512
mod-inventory-updatetags	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-updatetags:32.32.0	2	1024	896	128	768	88	128
mod-notifyauthtoken	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notifyauthtoken:32.215.01	2	10241440	8961152	128512	768922	88	128
modedge-configurationcourses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-configurationcourses:51.104.0	2	1024	896	128	768	88	128
mod-ordersinventory-storageupdate	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ordersinventory-storageupdate:133.73.0	2	1024	896	512128	700768	88	128
edgemod-caiasoftnotify	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-caiasoftnotify:23.2.0	2	1024	896	128	768	88	128
mod-login-samlconfiguration	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-samlconfiguration:25.810.0	2	1024	896	128	768	88	128
mod-ermorders-usage-harvesterstorage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ermorders-usage-harvesterstorage:413.57.0	2	1024	896	128512	768700	88	128
modedge-password-validatorcaiasoft	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-password-validatorcaiasoft:32.2.0	2	14401024	1298896	128	768	38488	512128
mod-login-licensessaml	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-licensessaml:62.08.0	2	24801024	2312896	128	1792768	38488	512128
mod-gobierm-usage-harvester	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8erm-usage-harvester:4.5.0	2	1024	896	128	700768	88	128
mod-bulkpassword-operationsvalidator	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulkpassword-operationsvalidator:3.2.0.0	2	30721440	26001298	1024128	1536768	384	512
mod-fqm-managerlicenses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-managerlicenses:26.0.10	2	30002480	26002312	128	20481792	384	512
edgemod-dcbgobi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-dcbgobi:12.18.0	2	1024	896	128	768700	88	128
mod-bulk-graphqloperations	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-graphqloperations:12.120.10	2	10243072	8962600	1281024	7681536	88384	128512
mod-financefqm-manager	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-financemanager:42.90.01	2	10243000	8962600	128	7002048	88384	128512
modedge-erm-usagedcb	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-erm-usagedcb:41.71.0	2	1024	896	128	768	88	128
mod-batch-printgraphql	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-printgraphql:1.12.1.0	2	1024	896	128	768	88	128
mod-copycatfinance	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycatfinance:14.69.0	2	1024	512896	128	768700	88	128
mod-erm-listsusage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-listsusage:24.07.0	2	30001024	2600896	128	2048768	38488	512128
mod-entitiesbatch-linksprint	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entitiesbatch-linksprint:31.01.0	2	25921024	2480896	400128	1440768	088	1024128
mod-permissionscopycat	21	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissionscopycat:1.6.5.0	2	16841024	1544512512	128	1024768	38488	512128
pubmod-edgelists	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pubmod-edgelists:20232.060.140	2	10243000	8962600	128	7682048	03840	512
mod-entities-orderslinks	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ordersentities-links:123.80.0	2	20482592	2480	400	1440	10240	1024	384	512
edge-patron	1mod-permissions	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-patronpermissions:6.5.1.0	2	10241684	8961544	256512	7681024	88384	128512
pub-edge-ncip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge-ncip:12023.906.214	2	1024	896	128	768	880	1280
mod-users-blorders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-blorders:712.78.0	2	2048	1440	11521024	5121024	922384	88512	128
modedge-invoicepatron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-invoicepatron:5.81.0	2	14401024	1152896	512256	922768	88	128
modedge-inventory-storagencip	21	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-inventory-storage:27.2.0-SNAPSHOT.738ncip:1.9.2	2	40961024	3690896	2048128	3076768	38488	512128
mod-userusers-importbl	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-userusers-importbl:37.87.0	2	10241440	8961152	128512	768922	88	128
mod-senderinvoice	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-senderinvoice:15.128.0	2	10241440	8961152	128512	768922	88	128
edgemod-oaiinventory-pmhstorage	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-oaiinventory-pmhstorage:27.2.90-SNAPSHOT.0738	2	15124096	13603690	10242048	14403076	384	512
mod-datauser-export-workerimport	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-datauser-export-workerimport:3.28.10	2	30721024	2048896	1024128	2048768	38488	512128
mod-rtacsender	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtacsender:31.612.0	2	1024	896	128	768	88	128
modedge-circulationoai-storagepmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-circulationoai-storagepmh:172.29.0	2	28801512	25921360	15361024	18141440	384	512
mod-sourcedata-recordexport-storageworker	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sourcedata-recordexport-storageworker:53.82.01	2	56003072	2048	50001024	20483500	384	512
mod-calendarrtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendarrtac:3.16.0	2	1024	896	128	768	88	128
mod-eventcirculation-configstorage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eventcirculation-configstorage:17.2.7.0	2	10242880	8962592	1281536	7681814	88384	128512
mod-source-record-coursesstorage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-coursesstorage:15.48.100	2	10245600	8965000	1282048	7683500	88384	128512
mod-circulation-itemcalendar	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-itemcalendar:3.1.0.0	2	1024	896	128	07680	88	0128
mod-inventoryevent-config	31	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-inventoryconfig:202.27.0	2	28801024	2592896	1024128	1814768	38488	512128
mod-emailcourses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-emailcourses:1.174.010	2	1024	896	128	768	88	128
mod-circulation-item	21	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:241.20.10	2	28801024	2592896	1536128	18140	3840	5120
mod-pubsubinventory	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsubinventory:20.2.13.0	2	15362880	14402592	1024	9221814	384	512
mod-di-converter-storage2email	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.2email:1.17.0	2	1024	896	128	768	88	128
edgemod-orderscirculation	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-orderscirculation:324.02.01	2	10242880	8962592	1281536	7681814	88384	128512
edgemod-rtacpubsub	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-rtacpubsub:2.713.10	2	10241536	8961440	1281024	768922	88384	128512
mod-users1di-converter-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-usersstorage:192.32.12	2	1024	896	128	768	88	128
modedge-template-engineorders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-template-engineorders:13.200.0	2	1024	896	128	768	88	128
modedge-patron-blocksrtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-patron-blocksrtac:12.107.01	2	1024	896	1024128	768	88	128
mod-auditusers	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-auditusers:219.93.01	2	1024	896	128	768	88	128
edgemod-template-fqmengine	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-template-fqmengine:21.020.0	2	1024	896	128	768	88	128
mod-sourcepatron-record-managerblocks	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sourcepatron-record-managerblocks:31.810.0	2	56001024	5000896	20481024	3500768	38488	512128
nginxmod-edgeaudit	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginxmod-edgeaudit:20232.069.140	2	1024	896	128	07680	88	0128
modedge-quick-marcfqm	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-quick-marcfqm:52.10.01	2	22881024	2176896	128	1664768	38488	512	nginx-okapi128
mod-source-record-manager	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14mod-source-record-manager:3.8.0	2	10245600	8965000	1282048	035000	384	0512
okapinginx-bedge	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapinginx-edge:52023.306.0143	2	16841024	1440896	1024128	9220	3840	5120
mod-quick-feesfinesmarc	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfinesquick-marc:195.1.0	21	10242288	8962176	128	7681664	88384	128512
modnginx-invoice-storageokapi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modnginx-invoice-storageokapi:52023.806.014	2	18721024	1536896	10241281024	0	3840	5120
modokapi-service-interactionb	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4okapi:5.3.0.1	23	20481684	18441440	2561024	1290922	384	512
mod-dcbfeesfines	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcbfeesfines:119.1.0	2	1024	896	128	768	88	128
mod-invoice-patronstorage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patroninvoice-storage:65.18.0	2	10241872	8961536	1281024	7681024	88384	128512
mod-dataservice-exportinteraction	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dataservice-exportinteraction:54.0.011	2	2048	15241844	1024256	012900	384	0512
mod-oai-pmhdcb	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmhdcb:31.131.0	2	40961024	3690896	2048128	3076768	38488	512128
edgemod-connexionpatron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-connexionpatron:6.1.2.0	2	1024	896	128	768	88	128
mod-notesdata-export	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-notesexport:5.20.02	1	10242048	8961524	1281024	9520	3840	5120
mod-kboai-ebsco-javapmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kboai-ebsco-javapmh:43.013.0	2	10244096	8963690	1282048	7683076	88384	128512
modedge-loginconnexion	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-loginconnexion:71.112.0	2	14401024	1298896	1024128	768	38488	512128
mod-organizations-storagenotes	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storagenotes:45.72.0	2	1024	896	128	700952	88384	128512
mod-datakb-exportebsco-springjava	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-datakb-exportebsco-springjava:34.20.01	2	20481024	1844896	256128	1536768	38488	512128
pubmod-okapilogin	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pubmod-okapilogin:20237.0611.140	2	10241440	8961298	1281024	768	03840	512
mod-eusageorganizations-reportsstorage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusageorganizations-reportsstorage:24.17.10	2	1024	896	128	768	88	128

Additional links and Errors

Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files.

09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []

Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and system stacked.

CPU Utilization when mod-permissions spiked and system stacked.

700	88	128
mod-data-export-spring	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
pub-okapi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Methodology/Approach

Data Export tests scenario using the profiles Default instances export job profile and srs - holdings and items were started from UI on Quesnelia (qcon) ecs environment.

...

Test 1: Manually tested 1k, 100k and 500k records files Data Export started on one College tenant(cs00000int_0001) only using Default instances export job profile.
Test 2: Manually tested 1k, 100k and 500k records files Data Export started on one College tenant(cs00000int_0001) only using srs - holdings and items job profile.
Test 3: Manually tested 1k, 100k and 500k records files Data Export started on central Central tenant(cs00000int) only using Default instances export job profile.
Test 4: Manually tested 1k, 100k and 500k records files Data Export started on central Central tenant(cs00000int) only using srs - holdings and items job profile.
Test 5: Manually tested 500k records file Data Export started on College tenant(cs00000int_0001), Central tenant(cs00000int) and Professional tenant(cs00000int_0002) concurrently using Default instances export job profile.

To get status and time range for export jobs the query used:

...

Version	Old Version 16	New Version Current
Changes made by	Stanislav Nehrii	Stanislav Nehrii
Saved on	Jun 01, 2024	Jun 03, 2024

Versions Compared

Key

Overview

Summary

Test Results

Comparison

Resource utilization for Test #1 and Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Overview

Summary

Test Results

Comparison

Resource utilization for Test #1 and Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

DB CPU Utilization

DB Connections

DB load

Appendix

Infrastructure

Resource utilization for Test #5

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Additional links and Errors

Methodology/Approach

#	TOP 5 SQL statements
1	`select mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$`
2	`INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING`
3	`select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)`
4	`select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1`
5	`select eie1_0.id,eie1_0.instance_id,_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.job_executioninstance_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only`

Page Comparison

Versions Compared

Key

Overview

Summary

Test Results

Comparison

Resource utilization for Test #1 and Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Overview

Summary

Test Results

Comparison

Resource utilization for Test #1 and Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

DB CPU Utilization

DB Connections

DB load

Appendix

Infrastructure

Resource utilization for Test #5

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Additional links and Errors

Methodology/Approach