Table of Contents

outline	true

Overview

This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-832

Summary

Test Results and Comparison

Test №1

Test with 1k, 10k, 25k and 50k records files DI started on one tenant only.

...

# of records

...

% creates

...

File

...

DI duration
Morning Glory

...

DI duration
Nolana

...

DI duration
Orchid

...

DI duration
Poppy

...

Test №2

Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K and 50K started on one tenant only.

Сomparative Data Import and Check-In\Check-Out results between Baseline and Quesnelia.

...

Test #

...

# of records

...

DI Duration

with CICO

...

CI time Avg
without

...

Baseline CI delta

...

CI time 95th pct

...

Baseline CI delta

...

CO time Avg

...

Baseline CO Avg

Delta

...

CO time 95th pct

...

Baseline CO delta

...

Сomparative Check-In\Check-Out results between Poppy and Quesnelia.

...

Table of Contents

outline	true

Overview

This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-832

Summary

Data import tests finished successfully, only Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files. Duration of DI grew in correspondence with the number of records in files.
Check-in and Check-out with 5 virtual users was performed during DI Create new MARC authority records jobs for non-matches No issues.
Data Import in Quesnelia without CICO perform faster than with it.
Comparing Poppy and Quesnelia releases
- Check-in / Check-out perform better in Quesnelia. Response time improved during Create jobs for long period of work time on 15% in Average.
- DI durations improved - 11%-14% in Average.
During testing, we noticed spikes in the mod permissions module. To mitigate this issue and prevent system slowdowns, we adjusted the order of loading files, starting with Tenant 3 (qcp1-02), followed by Tenant 2 (qcp1-01), and finally Tenant 1 (qcp1-00).

Test Results and Comparison

Test №1

Test with 1k, 10k, 25k and 50k records files DI started on one tenant only(qcp1-00), and comparative results between Poppy and Quesnelia.

# of records	% creates	File	DI duration Morning Glory	DI duration Nolana	DI duration Orchid	DI duration Poppy	DI duration Quesnelia
1,000	100	1k_marc_authority.mrc?api=v2	24 s	27 s	41 sec	29 sec	22 sec -24%
5,000	100	LC_SUBJ_msplit00000000.mrc?api=v2	1 min 21 s	1 min 15 s	1min 21s	1 min 38 sec	1 min 19 sec -19%
10,000	100	msplit00000000.mrc?api=v2	2 min 32 s	2 min 31 s	2min 53s	2 min 53 sec	2 min 36 sec -9.8%
22778 (for Poppy test) 25000 (for Quesnelia test)	100	msplit00000013.mrc?api=v2	11 min 14 s	7 min 7 s	5 min 42s	6 min 24 sec	6 min 19 sec -1.3%
50,000	100	50000_authorityrecords.mrc?api=v2	22 min	11 min 24 s	11 min 11s	13 min 48 sec	11 min 59 sec -13%

Test №2

Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K and 50K started on one tenant only.

Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.

	CICO, Median time without DI (Poppy)	CICO, 95% time without DI (Poppy)	CICO, Median time without DI (Quesnelia)	CICO, 95% time without DI (Quesnelia)	CICO, Avg time without DI (Quesnelia)
Check-In	516 ms	567 ms	503 ms -2.5%	593 ms +4.5%	511 ms
Check-Out	910 ms	2094 ms	836 ms -8%	1117 ms -46%	876 ms

Сomparative Check-In\Check-Out results between Baseline (Quesnelia) and Check-In\Check-Out plus Data Import (Quesnelia.)

# of records (Quesnelia)	DI Duration with CICO (Quesnelia)	CI time Avg (Quesnelia)	CI time 95th pct (Quesnelia)	CO time Avg (Quesnelia)	CO time 95th pct (Quesnelia)	Baseline CI Avg delta	Baseline CI 95th pct delta	Baseline CO Avg delta	Baseline CO 95th pct delta
1,000	20 sec	0.560	0.754	1.164	1.313	+9%	+27%	+32%	+17%
5,000	1 min 19 sec	0.701	1.171	1.141	1.790	+37%	+97%	+30%	+60%
10,000	2 min 35 se	0.723	1.024	1.179	1.494	+41%	+72%	+34%	+34%
25,000	6 min 26 sec	0.722	1.024	1.180	1.494	+41%	+72%	+35%	+34%
50,000	12 min 16 sec	0.777	1.045	1.265	1.550	+52%	+76%	+44%	+39%

Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.

# of records (Poppy)	DI Duration with CICO (Poppy)	CI time Avg (Poppy)	CI time 95th pct (Poppy)	CO time Avg (Poppy)	CO time 95th pct (Poppy)	# of records (Quesnelia)	DI Duration with CICO (Quesnelia)	CI time Avg (Quesnelia)	CI time 95th pct (Quesnelia)	CO time Avg (Quesnelia)	CO time 95th pct (Quesnelia)
1,000	35 sec	0.525	0.576	1.078	1.326	1,000	20 sec -42.8%	0.560 +6%	0.754 +30%	1.164 +8%	1.313 -1%
5,000	1 min 41 sec	0.513	0.612	0.9	1.019	5,000	1 min 19 sec -21.7%	0.701 +36%	1.171 +91%	1.141 +26%	1.790 +75%
10,000	3 min 4 sec	0.581	0.685	1.016	1.321	10,000	2 min 35 sec -15.7%	0.723 +24%	1.024 +49%	1.179 +16%	1.494 +13%
22,778	6 min 32 sec	0.598	1.542	1.244	1.729	25,000	6 min 26 sec -1.5%	0.722 +20%	1.024 -33%	1.180 -5%	1.494 -13%
50,000	13 min 48 sec	0.671	1.953	1.51	2.09	50,000	12 min 16 sec -11%	0.777 +15%	1.045 -46%	1.265 -16%	1.550 -25%

Test №3

Multitenant testing

...

Here we can see that mod-inventory-b module used 50% CPU and mod-source-record-storage-b 46% CPU

...

DB CPU in the average was 90%90%.

DB Connections

DB connections was 1151.

...


INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

...

1 database instances, writer

...

db.r6g.xlarge

...

4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=2

...

title	Quesnelia modules memory and CPU parameters

...

 (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer

Name Memory GIB vCPUs
db.r6g.xlarge
32 GiB 4 vCPUs
MSK ptf-mobius-testing2
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Expand

title	Quesnelia modules memory and CPU parameters

#277 Apr 26, 2024, 10:55 AM UTC
Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcp1-pvt
Fri Apr 26 10:56:03 UTC 2024
mod-remote-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512	FALSE
mod-ncip	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-finance-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128	FALSE
mod-agreements	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storageebsconet:32.2.0	2	49201248	447210241024	128	3960700	512128512	256	FALSE
mod-nciporganizations	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-nciporganizations:1.149.40	2	1024	896	128	768700	88	128	FALSE
mod-finance-storageconsortia	32	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storageconsortia:81.61.0	2	10243072	8962048	1024128	7002048	88512	1281024	FALSE
modedge-agreementssip2	32	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-agreementssip2:73.2.0-SNAPSHOT.0209	2	15921024	1488896	128	07680	88	0128	FALSE
mod-ebsconetserials-management	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-ebsconetmanagement:21.20.0	2	12482480	10242312	128	7001792	128384	256512	FALSE
mod-organizationssettings	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizationssettings:1.90.03	2	1024	896	128200	700768	88	128	FALSE
mod-data-consortiaimport	26	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-consortiaimport:13.1.0	21	2048	30721844	2048256	1281292	2048384	512	1024	FALSE
edge-sip2dematic	23	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2dematic:32.2.0-SNAPSHOT.2092	1	1024	896	128	768	88	128	FALSE
mod-serials-managementsearch	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-managementsearch:13.02.0	2	2592	2480	23122048	1281440	1792512	384	5121024	FALSE
mod-inn-settingsreach	32	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1inn-reach:3.2.0-SNAPSHOT.386	2	10243600	8963240	2001024	7682880	88512	1281024	FALSE
mod-data-importtags	63	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-importtags:32.12.01	2	20481024	1844896	256128	1292768	38488	512128	FALSE
edge-dematiccourses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematiccourses:21.24.01	2	1024	896	128	768	88	128	FALSE
mod-searchauthtoken	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-searchauthtoken:32.215.01	2	25921440	24801152	2048512	1440922	51288	1024128	FALSE
mod-inninventory-reachupdate	23	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inninventory-reachupdate:3.23.0-SNAPSHOT.86	2	36001024	3240896	1024128	2880768	51288	1024128	FALSE
mod-tagsnotify	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tagsnotify:23.2.0	2	1024	896	128	768	88	128	FALSE
edgemod-coursesconfiguration	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-coursesconfiguration:15.410.0	2	1024	896	128	768	88	128	FALSE
mod-orders-authtokenstorage	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-authtokenstorage:213.157.10	2	14401024	1152896	512	922700	88	128	FALSE
modedge-inventory-updatecaiasoft	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-inventory-updatecaiasoft:32.32.0	2	1024	896	128	768	88	128	FALSE
mod-login-notifysaml	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notifylogin-saml:32.28.0	2	1024	896	128	768	88	128	FALSE
mod-configurationerm-usage-harvester	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-configurationharvester:4.5.10.0	2	1024	896	128	768	88	128	FALSE
mod-orders-storagelicenses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storagelicenses:136.70.0	2	10242480	8962312	512128	7001792	88384	128512	FALSE
edgemod-caiasoftgobi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-caiasoftgobi:2.28.0	2	1024	896	128	768700	88	128	FALSE
mod-loginpassword-samlvalidator	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-loginpassword-samlvalidator:3.2.8.0	2	10241440	8961298	128	768	88384	128512	FALSE
mod-ermbulk-usage-harvesteroperations	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ermbulk-usage-harvesteroperations:42.50.0	2	10243072	8962600	1281024	7681536	88384	128512	FALSE
mod-fqm-licensesmanager	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-licensesmanager:62.0.01	2	24803000	23122600	128	17922048	384	512	FALSETRUE
modedge-gobidcb	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-gobidcb:21.81.0	2	1024	896	128	700768	88	128	FALSE
mod-password-validatorgraphql	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validatorgraphql:31.212.01	2	14401024	1298896	128	768	38488	512128	FALSE
mod-bulk-operationsfinance	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operationsfinance:24.09.0	2	30721024	2600896	1024128	1536700	38488	512128	FALSE
mod-fqmerm-managerusage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqmerm-managerusage:24.7.0.1	2	30001024	2600896	128	2048768	38488	512128	TRUEFALSE
edgemod-batch-dcbprint	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-batch-dcbprint:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-graphqlcopycat	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphqlcopycat:1.126.10	2	1024	896512	128	768	88	128	FALSE
mod-financelists	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-financelists:42.90.0	2	10243000	8962600	128	7002048	88384	128512	FALSE
mod-ermentities-usagelinks	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ermentities-usagelinks:43.70.0	2	10242592	8962480	128400	7681440	880	1281024	FALSE
mod-batch-printpermissions	46	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-printpermissions:16.15.0	2	10241684	8961544	128512	7681024	88384	128512	FALSE
modpub-copycatedge	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modpub-copycatedge:12023.606.014	2	1024	512896	128	768	880	1280	FALSE
mod-listsorders	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-listsorders:212.08.0	2	30002048	26001440	1281024	20481024	384	512	FALSE
modedge-entities-linkspatron	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-entities-linkspatron:35.01.0	2	25921024	2480896	400256	1440768	088	1024128	FALSE
modedge-permissionsncip	63	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-permissionsncip:61.59.02	2	16841024	1544896	512128	1024768	38488	512128	FALSE
pubedge-inn-edgereach	32	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pubedge-inn-edgereach:2023.06.143.1.1-SNAPSHOT.45	2	1024	896	128	768	0880	128	FALSE
mod-users-ordersbl	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-ordersbl:127.87.0	2	204814401440	1152	10245121024	922	38488	512128	FALSE
edgemod-patronoa	32	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-patronoa:52.1.0-SNAPSHOT.62	2	1024	896	256128	768	88	128	FALSE
edgemod-inventory-ncipstorage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-inventory-ncipstorage:27.1.9.20	2	10244096	8963690	1282048	7683076	88384	128512	FALSE
edgemod-inn-reachinvoice	24	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-inn-reachinvoice:35.1.1-SNAPSHOT.458.0	2	10241440	8961152	128512	768922	88	128	FALSE
mod-usersuser-blimport	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-usersuser-blimport:73.78.0	2	14401024	1152896	512128	922768	88	128	FALSE
mod-oasender	24	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oasender:21.112.0-SNAPSHOT.62	2	1024	896	128	768	88	128	FALSE
modedge-inventoryoai-storagepmh	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-inventoryoai-storagepmh:272.19.0	2	40961512	36901360	20481024	30761440	384	512	FALSE
mod-data-invoiceexport-worker	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-invoiceworker:53.82.01	2	14403072	11522048	5121024	9222048	88384	128512	FALSE
mod-user-importrtac	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-importrtac:3.86.0	2	1024	896	128	768	88	128	FALSE
mod-sendercirculation-storage	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sendercirculation-storage:117.122.0	2	10242880	8962592	1281536	7681814	88384	128512	FALSE
edgemod-oai-pmhcalendar	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-oai-pmhcalendar:23.91.0	2	15121024	1360896	1024128	1440768	38488	512128	FALSE
mod-datasource-exportrecord-workerstorage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-datasource-exportrecord-workerstorage:35.28.10	2	5600	30725000	2048	1024	20483500	384	512	FALSE
mod-event-rtacconfig	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-rtacconfig:32.67.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storagecourses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storagecourses:171.24.010	2	28801024	2592896	1536128	1814768	38488	512128	FALSE
mod-circulation-calendaritem	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendarcirculation-item:31.10.0	2	1024	896	128	7680	880	1280	FALSE
mod-source-record-storageinventory	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8inventory:20.2.0	2	56002880	50002592	20481024	35001814	384	512	FALSE
mod-event-configemail	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-configemail:21.717.0	2	1024	896	128	768	88	128	FALSE
mod-coursespubsub	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-coursespubsub:12.413.100	2	10241536	8961440	1281024	768922	88384	128512	FALSE
mod-circulation-item	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:124.02.0	2	10242880	8962592	1281536	018140	384	0512	FALSE
mod-di-converter-inventorystorage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-inventoryconverter-storage:202.2.0	2	28801024	2592896	1024128	1814768	38488	512128	FALSE
modedge-emailrtac	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-emailrtac:12.177.01	2	1024	896	128	768	88	128	FALSE
modedge-pubsuborders	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-pubsuborders:23.130.0	2	15361024	1440896	1024128	922768	38488	512128	FALSE
mod-circulationusers	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulationusers:2419.23.01	2	28801024	2592896	1536128	1814768	38488	512128	FALSE
mod-ditemplate-converter-storageengine	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ditemplate-converter-storageengine:21.220.0	2	1024	896	128	768	88	128	FALSE
edgemod-patron-rtacblocks	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-patron-rtacblocks:21.710.10	2	1024	896	1281024	768	88	128	FALSE
edgemod-ordersaudit	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-ordersaudit:32.09.0	2	1024	896	128	768	88	128	FALSE
modedge-usersfqm	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-usersfqm:192.30.10	2	1024	896	128	768	88	128	FALSE
mod-templatesource-record-enginemanager	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-templaterecord-enginemanager:13.208.0	2	10245600	8965000	1282048	7683500	88384	128512	FALSE
modnginx-patron-blocksedge	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modnginx-patron-blocksedge:12023.1006.014	2	1024	896	1024128	7680	880	1280	FALSE
mod-quick-auditmarc	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-auditquick-marc:25.91.02	1	10242288	8962176	128	7681664	88384	128512	FALSE
edgenginx-fqmokapi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgenginx-fqmokapi:22023.006.014	2	1024	896	128	7680	880	1280	FALSE
mod-source-record-manager	3okapi-b	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8okapi:5.3.0	23	56001684	50001440	20481024	3500922	384	512	FALSE
nginxmod-edgefeesfines	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginxmod-edgefeesfines:202319.061.140	2	1024	896	128	07680	88	0128	FALSE
mod-quickinvoice-marcstorage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quickinvoice-marcstorage:5.18.01	2	22881872	21761536	1281024	16641024	384	512	FALSE
nginxmod-okapidcb	34	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginxmod-okapidcb:20231.061.140	2	1024	896	128	07680	88	0128	FALSE
okapimod-service-binteraction	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.0	3	1684	1440	1024	922mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512	FALSE
mod-feesfinesdata-export	310	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-feesfinesexport:195.10.04	21	10242048	8961524	1281024	7680	880	1280	FALSE
mod-invoice-storagepatron	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storagepatron:56.81.0	2	18721024	1536896	10241281024	768	38488	512128	FALSE
mod-dcboai-pmh	43	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-dcbpmh:13.113.0	2	10244096	8963690	1282048	7683076	88384	128512	FALSE
modedge-service-interactionconnexion	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-service-interaction:4connexion:1.2.0.1	2	20481024	1844896	256128	1290768	38488	512128	FALSE
mod-kb-dataebsco-exportjava	103	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-dataebsco-exportjava:54.0.401	2	20481024	1524896	1024128	07680	88	0128	FALSE
mod-patronnotes	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patronnotes:65.12.0	2	1024	896	128	768952	88384	128512	FALSE
mod-oaidata-export-pmhspring	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-oaiexport-pmhspring:3.132.02	1	40962048	36901844	2048256	30761536	384	512	FALSE
edgemod-organizations-connexionstorage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-organizations-connexionstorage:14.27.0	2	1024	896	128	768700	88	128	FALSE
mod-kb-ebsco-javalogin	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0login:7.11.0	2	1440	1298	1024	896	128	768	88384	128512	FALSE
modpub-notesokapi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modpub-notesokapi:52023.206.014	2	1024	896	128	952768	3840	5120	FALSE
mod-dataeusage-export-springreports	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dataeusage-export-spring:3.2.0	1	2048	1844	256	1536	384	512	FALSE
mod-organizations-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128	FALSE
mod-login	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.0	2	1440	1298	1024	768	384	512	FALSE
pub-okapi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128	FALSE
reports:2.1.1	2	1024	896	128	768	88	128	FALSE

Additional links and Errors

Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files.

09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []

Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and system stacked.

CPU Utilization when mod-permissions spiked and system stacked.

Image Added

CPU Utilization when mod-permissions spiked and system stacked.

Image Added

Recommendations & Jiras (Optional)

Link to Jira ticket: https://folio-org.atlassian.net/browse/PERF-801

Methodology/Approach

DI tests scenario a data import job profile that creates new MARC authority records for non-matches (Job Profile: KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q) were started from UI on Quesnelia (qcp1) env with file splitting features enabled on a non-ecs environment..

...

The above files are all stored here - MARC Resources
- 22k file what was provided from MARC Resources does nor work, so 50k file was split to file with 25k records and used instead of 22k file.
At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl file, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).

Test set

Test 1: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only.
Test 2: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only plus Check-in and Checkout (CICO) for 5 concurrent users.
Test 3: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)
Test 4: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file with pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)
Test 5: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 1k, 5k, 10k, 25k and 50k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)

...

Version	Old Version 41	New Version Current
Changes made by	Stanislav Nehrii	Stanislav Nehrii
Saved on	May 04, 2024	May 27, 2024

#	TOP 5 SQL statements
1	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
2	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
3	`INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
4	`INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
5	`INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`

Versions Compared

Key

Overview

Summary

Test Results and Comparison

Overview

Summary

Test Results and Comparison

DB Connections

Appendix

Infrastructure

Appendix

Infrastructure

Additional links and Errors

Recommendations & Jiras (Optional)

Methodology/Approach

Name	Memory GIB	vCPUs
db.r6g.xlarge	32 GiB	4 vCPUs

Page Comparison

Versions Compared

Key

Overview

Summary

Test Results and Comparison

Overview

Summary

Test Results and Comparison

DB Connections

Appendix

Infrastructure

Appendix

Infrastructure

Additional links and Errors

Recommendations & Jiras (Optional)

Methodology/Approach