Overview

This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

PERF-832 - Getting issue details... STATUS

Summary

Test Results and Comparison

Test №1

Test with 1k, 10k, 25k and 50k records files DI started on one tenant only.

# of records	% creates	File	DI duration Morning Glory	DI duration Nolana	DI duration Orchid	DI duration Poppy	DI duration Quesnelia
1,000	100	1k_marc_authority.mrc?api=v2	24 s	27 s	41 sec	29 sec	22 sec -24%
5,000	100	LC_SUBJ_msplit00000000.mrc?api=v2	1 min 21 s	1 min 15 s	1min 21s	1 min 38 sec	1 min 19 sec -19%
10,000	100	msplit00000000.mrc?api=v2	2 min 32 s	2 min 31 s	2min 53s	2 min 53 sec	2 min 36 sec -9.8%
22778 (for Poppy test) 25000 (for Quesnelia test)	100	msplit00000013.mrc?api=v2	11 min 14 s	7 min 7 s	5 min 42s	6 min 24 sec	6 min 19 sec -1.3%
50,000	100	50000_authorityrecords.mrc?api=v2	22 min	11 min 24 s	11 min 11s	13 min 48 sec	11 min 59 sec -13%

Test №2

Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K and 50K started on one tenant only.

Сomparative Data Import and Check-In\Check-Out results between Baseline and Quesnelia.

# of records	DI Duration with CICO	CI time Avg without	Baseline CI delta	CI time 95th pct	Baseline CI delta	CO time Avg	Baseline CO Avg Delta	CO time 95th pct	Baseline CO delta
1,000	35 sec	0.525	+2%	0.576	+1%	1.078	+18%	1.326	-27%
5,000	1 min 41 sec	0.513	-1%	0.612	+8%	0.9	-0.001%	1.019	-52%
10,000	3 min 4 sec	0.581	+12%	0.685	+20%	1.016	+11%	1.321	-27%
22,778	6 min 32 sec	0.598	+16%	1.542	+171%	1.244	+36%	1.729	-18%
50,000	13 min 48 sec	0.671	+30%	1.953	+244%	1.51	+65%	2.09	-0.01%

Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.

	CICO, Median time without DI (Poppy)	CICO, 95% time without DI (Poppy)	CICO, Median time without DI (Quesnelia)	CICO, 95% time without DI (Quesnelia)	CICO, Avg time without DI (Quesnelia)
Check-In	516 ms	567 ms	503 ms -2.5%	593 ms +4.5%	511 ms
Check-Out	910 ms	2094 ms	836 ms -8%	1117 ms -46%	876 ms

Сomparative Check-In\Check-Out results between Baseline (Quesnelia) and Check-In\Check-Out plus Data Import (Quesnelia.)

# of records (Quesnelia)	DI Duration with CICO (Quesnelia)	CI time Avg (Quesnelia)	CI time 95th pct (Quesnelia)	CO time Avg (Quesnelia)	CO time 95th pct (Quesnelia)	Baseline CI Avg delta	Baseline CI 95th pct delta	Baseline CO Avg delta	Baseline CO 95th pct delta
1,000	20 sec	0.560	0.754	1.164	1.313	+9%	+27%	+32%	+17%
5,000	1 min 19 sec	0.701	1.171	1.141	1.790	+37%	+97%	+30%	+60%
10,000	2 min 35 se	0.723	1.024	1.179	1.494	+41%	+72%	+34%	+34%
25,000	6 min 26 sec	0.722	1.024	1.180	1.494	+41%	+72%	+35%	+34%
50,000	12 min 16 sec	0.777	1.045	1.265	1.550	+52%	+76%	+44%	+39%

Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.

# of records (Poppy)	DI Duration with CICO (Poppy)	CI time Avg (Poppy)	CI time 95th pct (Poppy)	CO time Avg (Poppy)	CO time 95th pct (Poppy)	# of records (Quesnelia)	DI Duration with CICO (Quesnelia)	CI time Avg (Quesnelia)	CI time 95th pct (Quesnelia)	CO time Avg (Quesnelia)	CO time 95th pct (Quesnelia)
1,000	35 sec	0.525	0.576	1.078	1.326	1,000	20 sec -42.8%	0.560 +6%	0.754 +30%	1.164 +8%	1.313 -1%
5,000	1 min 41 sec	0.513	0.612	0.9	1.019	5,000	1 min 19 sec -21.7%	0.701 +36%	1.171 +91%	1.141 +26%	1.790 +75%
10,000	3 min 4 sec	0.581	0.685	1.016	1.321	10,000	2 min 35 sec -15.7%	0.723 +24%	1.024 +49%	1.179 +16%	1.494 +13%
22,778	6 min 32 sec	0.598	1.542	1.244	1.729	25,000	6 min 26 sec -1.5%	0.722 +20%	1.024 -33%	1.180 -5%	1.494 -13%
50,000	13 min 48 sec	0.671	1.953	1.51	2.09	50,000	12 min 16 sec -11%	0.777 +15%	1.045 -46%	1.265 -16%	1.550 -25%

Test №3

Multitenant testing

File with 50K, 25K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3 for Poppy

File with 50K, 25K, 10K, 5K, 1K loaded sequences and launched on Tenant3, Tenant2 and Tenant1 for Quesnelia

Num of records	Tenant 1 pcp1-00 duration (Poppy)	Tenant 2 pcp1-01 duration (Poppy)	Tenant 3 pcp1-02 duration (Poppy)	Num of records	Tenant 3 qcp1-02 duration (Quesnelia)	Tenant 2 qcp1-01 duration (Quesnelia)	Tenant 1 qcp1-00 duration (Quesnelia)
1000	66 min	75 min	74 min	1000	15 min 15 sec	15 min 46 sec	16 min
5000	67 min	76 min	75 min	5000	22 min 9 sec	22 min 10 sec	21 min 33 sec
10,000	67 min	76 min	74 min	10,000	27 min 38 sec	27 min 51 sec	29 min 42 sec
22,778	67 min*	73 min*	71 min*	25000	46 min 42 sec	47 min	47 min 14 sec
50,000	62 min	56 min	54 min	50,000	67 min 29 sec	67 min 27 sec	67 min 22 sec

Test №4

Order for load files with pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)

Job Profile	Num of records	Tenant 1 qcp1-00 duration	Tenant 2 qcp1-01 duration	Tenant 3 qcp1-02 duration
KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q	1000	00:28	00:32	00:18
	5000	03:38	03:31	03:29
	10,000	06:58	07:02	06:47
	25000	17:54	17:35	17:53
	50,000	34:09	35:08	29:49

Test №5

Order for load files without pause between files: 1k, 5k, 10k, 25k and 50k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)

Job Profile	Num of records	Tenant 1 qcp1-00 duration	Tenant 2 qcp1-01 duration	Tenant 3 qcp1-02 duration
KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q	1000	0:09:37	0:02:43	0:00:53
	5000	0:10:54	0:28:43	0:02:17
	10,000	0:19:13	0:31:45	0:05:53
	25000	0:35:54	0:44:13	0:22:44
	50,000	1:03:55	1:10:16	0:52:38

Resource utilization for Test #1

Resource utilization table

CPU		RAM
mod-inventory-b	49.90%	mod-data-import-b	64.80%
mod-source-record-storage-b	44.70%	mod-permissions-b	63.50%
nginx-okapi	38.10%	mod-source-record-manager-b	49.90%
mod-di-converter-storage-b	28.40%	mod-inventory-b	41.80%
mod-source-record-manager-b	24.80%	mod-di-converter-storage-b	41.30%
okapi-b	16.40%	okapi-b	36.20%
mod-data-import-b	16.10%	mod-source-record-storage-b	27.70%
mod-permissions-b	5.60%	mod-inventory-storage-b	16.20%
mod-inventory-storage-b	0.50%	nginx-okapi	4.90%
pub-okapi	0.20%	pub-okapi	4.70%

Service CPU Utilization

Here we can see that mod-inventory-b module used 50% CPU and mod-source-record-storage-b 46% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU in the average was 90%.

DB Connections

DB connections was 1151.

DB load

Top SQL-queries


INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
WITH input_rows(record_id, authority_id) AS (
   VALUES ($1::uuid,$2::uuid)
)
, ins AS (
   INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id)
   SELECT * FROM input_rows
   ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
   RETURNING record_id::uuid, authority_id::uuid
   )
SELECT record_id, authority_id
FROM   ins
UNION  ALL
SELECT c.record_id, c.authority_id 
FROM   input_rows
JOIN   fs09000000_mod_inventory.records_authorities c USING (record_id);
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

Resource utilization for Test #2

Resource utilization table

CPU		RAM
nginx-okapi	49%	mod-inventory-b	75%
mod-inventory-b	47%	mod-permissions-b	70%
mod-source-record-storage-b	42%	mod-data-import-b	61%
mod-di-converter-storage-b	29%	mod-source-record-manager-b	48%
mod-source-record-manager-b	26%	okapi-b	36%
okapi-b	22%	mod-inventory-storage-b	35%
mod-permissions-b	7%	mod-di-converter-storage-b	34%
pub-okapi	2%	mod-source-record-storage-b	28%
mod-inventory-storage-b	2%	nginx-okapi	5%
mod-data-import-b	2%	pub-okapi	4%

Service CPU Utilization

Here we can see that mod-inventory-b module and nginx-okapi used 50% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 93%.

DB Connections

DB connections was 1580.

DB load

Top SQL-queries


INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
WITH input_rows(record_id, authority_id) AS (
   VALUES ($1::uuid,$2::uuid)
)
, ins AS (
   INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id)
   SELECT * FROM input_rows
   ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
   RETURNING record_id::uuid, authority_id::uuid
   )
SELECT record_id, authority_id
FROM   ins
UNION  ALL
SELECT c.record_id, c.authority_id 
FROM   input_rows
JOIN   fs09000000_mod_inventory.records_authorities c USING (record_id);

Resource utilization for Test #3

Resource utilization table

CPU		RAM
mod-permissions-b	146%	mod-permissions-b	79%
mod-inventory-b	56%	mod-inventory-b	72%
nginx-okapi	42%	mod-data-import-b	61%
mod-source-record-storage-b	41%	mod-source-record-manager-b	44%
mod-di-converter-storage-b	37%	okapi-b	37%
mod-source-record-manager-b	27%	mod-source-record-storage-b	36%
okapi-b	20%	mod-inventory-storage-b	36%
mod-data-import-b	2%	mod-di-converter-storage-b	35%
mod-inventory-storage-b	1%	nginx-okapi	5%
pub-okapi	1%	pub-okapi	4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 57% CPU, mod-inventory-b used 43% CPU and mod-permissions-b spiked up to 150% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 95%.

DB Connections

DB connections was 1823.

DB load

Top SQL-queries


INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Resource utilization for Test #4

Resource utilization table

CPU		RAM
mod-inventory-b	91.30%	mod-permissions-b	80.00%
nginx-okapi	43.60%	mod-data-import-b	65.10%
mod-source-record-storage-b	39.90%	mod-inventory-b	60.40%
mod-di-converter-storage-b	26.60%	mod-source-record-storage-b	42.20%
okapi-b	25.00%	mod-di-converter-storage-b	42.00%
mod-source-record-manager-b	18.90%	mod-source-record-manager-b	41.90%
mod-data-import-b	2.40%	okapi-b	37.60%
mod-permissions-b	1.60%	mod-inventory-storage-b	16.30%
mod-inventory-storage-b	0.30%	nginx-okapi	5.60%
pub-okapi	0.20%	pub-okapi	4.80%

Service CPU Utilization

Here we can see that mod-inventory-b module used 91% CPU, nginx-okapiused 44% CPU and mod-permissions-b spiked up to used 170% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 90%.

DB Connections

DB connections was 1450.

DB load

Top SQL-queries


INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Resource utilization for Test #5

Resource utilization table

CPU		RAM
mod-permissions-b	77%	mod-permissions-b	77%
mod-inventory-b	63%	mod-inventory-b	75%
mod-di-converter-storage-b	35%	mod-data-import-b	59%
nginx-okapi	35%	mod-source-record-manager-b	53%
mod-source-record-storage-b	35%	mod-source-record-storage-b	44%
mod-source-record-manager-b	23%	mod-di-converter-storage-b	34%
mod-data-import-b	20%	mod-inventory-storage-b	34%
okapi-b	19%	okapi-b	33%
mod-inventory-storage-b	1%	nginx-okapi	4%
pub-okapi	1%	pub-okapi	4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 63% CPU in avarage, and mod-permissions-b spiked up to 130% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 90%.

DB Connections

DB connections was 1757.

DB load

Top SQL-queries


INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer
Name Memory GIB vCPUs
db.r6g.xlarge
32 GiB 4 vCPUs
MSK ptf-mobius-testing2
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Quesnelia modules memory and CPU parameters

#277 Apr 26, 2024, 10:55 AM UTC
Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcp1-pvt
Fri Apr 26 10:56:03 UTC 2024
mod-remote-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512	FALSE
mod-ncip	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-finance-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128	FALSE
mod-agreements	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256	FALSE
mod-organizations	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.0	2	1024	896	128	700	88	128	FALSE
mod-consortia	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024	FALSE
edge-sip2	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	128	768	88	128	FALSE
mod-serials-management	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512	FALSE
mod-settings	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.3	2	1024	896	200	768	88	128	FALSE
mod-data-import	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.0	1	2048	1844	256	1292	384	512	FALSE
edge-dematic	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.0	1	1024	896	128	768	88	128	FALSE
mod-search	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.2.0	2	2592	2480	2048	1440	512	1024	FALSE
mod-inn-reach	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	1024	2880	512	1024	FALSE
mod-tags	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.0	2	1024	896	128	768	88	128	FALSE
edge-courses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.4.0	2	1024	896	128	768	88	128	FALSE
mod-authtoken	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.15.1	2	1440	1152	512	922	88	128	FALSE
mod-inventory-update	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.3.0	2	1024	896	128	768	88	128	FALSE
mod-notify	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.2.0	2	1024	896	128	768	88	128	FALSE
mod-configuration	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.10.0	2	1024	896	128	768	88	128	FALSE
mod-orders-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.7.0	2	1024	896	512	700	88	128	FALSE
edge-caiasoft	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.2.0	2	1024	896	128	768	88	128	FALSE
mod-login-saml	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.8.0	2	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128	FALSE
mod-licenses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.0.0	2	2480	2312	128	1792	384	512	FALSE
mod-gobi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8.0	2	1024	896	128	700	88	128	FALSE
mod-password-validator	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.2.0	2	1440	1298	128	768	384	512	FALSE
mod-bulk-operations	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512	FALSE
mod-fqm-manager	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512	TRUE
edge-dcb	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-graphql	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.1	2	1024	896	128	768	88	128	FALSE
mod-finance	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.9.0	2	1024	896	128	700	88	128	FALSE
mod-erm-usage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.7.0	2	1024	896	128	768	88	128	FALSE
mod-batch-print	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-copycat	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.6.0	2	1024	512	128	768	88	128	FALSE
mod-lists	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:2.0.0	2	3000	2600	128	2048	384	512	FALSE
mod-entities-links	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024	FALSE
mod-permissions	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.5.0	2	1684	1544	512	1024	384	512	FALSE
pub-edge	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-orders	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.8.0	2	2048	1440	1024	1024	384	512	FALSE
edge-patron	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.1.0	2	1024	896	256	768	88	128	FALSE
edge-ncip	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.2	2	1024	896	128	768	88	128	FALSE
edge-inn-reach	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	128	768	88	128	FALSE
mod-users-bl	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.7.0	2	1440	1152	512	922	88	128	FALSE
mod-oa	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	128	768	88	128	FALSE
mod-inventory-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512	FALSE
mod-invoice	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.8.0	2	1440	1152	512	922	88	128	FALSE
mod-user-import	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.0	2	1024	896	128	768	88	128	FALSE
mod-sender	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.12.0	2	1024	896	128	768	88	128	FALSE
edge-oai-pmh	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512	FALSE
mod-rtac	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.6.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512	FALSE
mod-calendar	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512	FALSE
mod-event-config	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.7.0	2	1024	896	128	768	88	128	FALSE
mod-courses	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.10	2	1024	896	128	768	88	128	FALSE
mod-circulation-item	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.0	2	1024	896	128	0	0	0	FALSE
mod-inventory	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512	FALSE
mod-email	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.17.0	2	1024	896	128	768	88	128	FALSE
mod-pubsub	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512	FALSE
mod-circulation	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.2.0	2	2880	2592	1536	1814	384	512	FALSE
mod-di-converter-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128	FALSE
edge-rtac	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.7.1	2	1024	896	128	768	88	128	FALSE
edge-orders	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.0.0	2	1024	896	128	768	88	128	FALSE
mod-users	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.1	2	1024	896	128	768	88	128	FALSE
mod-template-engine	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.20.0	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128	FALSE
mod-audit	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.9.0	2	1024	896	128	768	88	128	FALSE
edge-fqm	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:2.0.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512	FALSE
nginx-edge	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.14	2	1024	896	128	0	0	0	FALSE
mod-quick-marc	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.0	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.1.0	2	1024	896	128	768	88	128	FALSE
mod-invoice-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512	FALSE
mod-dcb	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-service-interaction	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512	FALSE
mod-data-export	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.0.4	1	2048	1524	1024	0	0	0	FALSE
mod-patron	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.1.0	2	1024	896	128	768	88	128	FALSE
mod-oai-pmh	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512	FALSE
edge-connexion	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.2.0	2	1024	896	128	768	88	128	FALSE
mod-kb-ebsco-java	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	FALSE
mod-notes	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.2.0	2	1024	896	128	952	384	512	FALSE
mod-data-export-spring	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512	FALSE
mod-organizations-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128	FALSE
mod-login	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.0	2	1440	1298	1024	768	384	512	FALSE
pub-okapi	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128	FALSE

Additional links and Errors

Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files.

09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []

Link to Jira ticket: https://folio-org.atlassian.net/browse/PERF-801

Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and system stacked.

CPU Utilization when mod-permissions spiked and system stacked.

Methodology/Approach

DI tests scenario a data import job profile that creates new MARC authority records for non-matches (Job Profile: KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q) were started from UI on Quesnelia (qcp1) env with file splitting features enabled on a non-ecs environment..

Action for non-matches: Create MARC authority record

The above files are all stored here - MARC Resources
- 22k file what was provided from MARC Resources does nor work, so 50k file was split to file with 25k records and used instead of 22k file.
At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl file, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).

Test set

Test 1: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant only.
Test 2: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant only plus Check-in and Checkout (CICO) for 5 concurrent users.
Test 3: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)
Test 4: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file with pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)
Test 5: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 1k, 5k, 10k, 25k and 50k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)

#	TOP 5 SQL statements
1	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
2	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
3	`insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)`
4	`WITH input_rows(record_id, authority_id) AS ( VALUES ($1::uuid,$2::uuid) ) , ins AS ( INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id) SELECT * FROM input_rows ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id RETURNING record_id::uuid, authority_id::uuid ) SELECT record_id, authority_id FROM ins UNION ALL SELECT c.record_id, c.authority_id FROM input_rows JOIN fs09000000_mod_inventory.records_authorities c USING (record_id);`
5	`UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *`

#	TOP 5 SQL statements
1	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
2	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
3	`UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *`
4	`insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)`
5	`WITH input_rows(record_id, authority_id) AS ( VALUES ($1::uuid,$2::uuid) ) , ins AS ( INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id) SELECT * FROM input_rows ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id RETURNING record_id::uuid, authority_id::uuid ) SELECT record_id, authority_id FROM ins UNION ALL SELECT c.record_id, c.authority_id FROM input_rows JOIN fs09000000_mod_inventory.records_authorities c USING (record_id);`

#	TOP 5 SQL statements
1	`INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
2	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
3	`INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
4	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
5	`INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`

#	TOP 5 SQL statements
1	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
2	`INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
3	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
4	`INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
5	`INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`

#	TOP 5 SQL statements
1	`INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
2	`INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
3	`INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)`
4	`INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`
5	`INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)`

Name	Memory GIB	vCPUs
db.r6g.xlarge	32 GiB	4 vCPUs

[Quesnelia] [non-ECS] [Data import] Create MARC authority Records

Overview

Summary

Test Results and Comparison

Resource utilization for Test #1

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #3

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #4

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #5

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Additional links and Errors

Methodology/Approach