Overview IN PROGRESS

This document contains the results of testing workflows Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release with a new MSK instance type. The main idea is to see how the m7g series affects FOLIO performance. Compared results for main workflows with different instance types: kafka.m5.2xlarge against kafka.m7g.2xlarge.

Ticket: PERF-921 - Getting issue details... STATUS

Summary

Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
- Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.
Resource utilization
- Memory utilization didn't differ a lot between two MSK clusters
- Disk consume less resources in both (idle and CICO+DI) scenarios with m7g instance type. CPU utilization the same in idle state and from 3% to 18% less under load with m7g instance type.
- Delta for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The most part of modules CPU utilization fluctuate under 10%.
- Average DB CPU usage for both MSK clusters during data import is 85% during create jobs and 87& during Update jobs. Check-In/Check-Out period without DI - 15%.
- Average connection count for both MSK clusters during data import is about 850 connections for create and update jobs with CI/CO. And 730 connections for CI/CO without data import
- MSK instance CPU and Disk utilization remain on the same level in kafka.m5.2xlarge against kafka.m7g.2xlarge or even decreased.

Test Runs

Test #	MSK instance type	Scenario	Load level
1	kafka.m5.2xlarge	CICO + DI MARC Bib Create	8 users + 5K, 25K sequentially
2		DI MARC Bib Create	5K, 25K sequentially
3		CICO + DI MARC Bib Update	8 users + 5K, 25K sequentially
4		DI MARC Bib Update	5K, 25K sequentially
5	kafka.m7g.2xlarge	CICO + DI MARC Bib Create	8 users + 5K, 25K sequentially
6		DI MARC Bib Create	5K, 25K sequentially
7		CICO + DI MARC Bib Update	8 users + 5K, 25K sequentially
8		DI MARC Bib Update	5K, 25K sequentially

Test Results

This table shows results of Check-In/Check-out and Data Import create and update jobs.

The only difference between tests is MSK cluster instance type. Cluster ptf-mobius-testing2 has kafka.m5.2xlarge and cluster PERF-921 has kafka.m7g.2xlarge

MSK instance: kafka.m5.2xlarge	File size	DI Duration without CI/CO	DI Duration	CI Average sec	CO Average sec
Create	5k	00:02:31	00:02:54	0.899	1.409
	25k	00:11:49	00:12:49	0.724	1.152
Update	5k	00:03:06	00:03:14	0.807	1.257
	25k	00:15:00	00:15:30	0.784	1.275
MSK instance: kafka.m7g.2xlarge
Create	5k	00:03:05	00:02:39	0.707	1.104
	25k	00:12:03	00:12:08	0.718	1.129
Update	5k	00:03:36	00:03:34	0.742	1.124
	25k	00:17:05	00:17:33	0.756	1.148

Check-in/Check-out without DI

Scenario	Load level	Request	Response time, sec MSK instance: kafka.m5.2xlarge		Response time, sec MSK instance: kafka.m7g.2xlarge
Scenario	Load level	Request	95 perc	average	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.669	0.570	0.720	0.606
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1.152	0.960	1.241	0.969

Comparison

Data Import durations and Check-In/Check-Out response time comparison

Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.

Job Profile	File size	DELTA, DI	DELTA, DI+CICO	DELTA, CI	DELTA, CO
PTF - Create 2	5k	00:00:34	00:00:15	0.192	0.305
PTF - Create 2	25k	00:00:14	00:00:41	0.006	0.023
PTF - Updates Success - 6	5k	00:00:31	00:00:20	0.065	0.133
PTF - Updates Success - 6	25k	00:02:06	00:02:03	0.028	0.127

Kafka resource utilization comparison table

Disk consume less resources in both (idle and CICO+DI) scenarios with m7g instance type. CPU utilization the same in idle state and from 3% to 18% less under load with m7g instance type.

Utilized resources	Scenario/Load	Brokers	MSK instance: kafka.m5.2xlarge	MSK instance: kafka.m7g.2xlarge	Delta
Disk usage	Idle state
		1	0.474	0.356	-0.118
		2	0.476	0.357	-0.119
	CICO+DI (update 25k)
		1	4.6120204	4.35	-0.26202
		2	4.611104	4.35	-0.2611
CPU usage	Idle state
		1	6.431249875	6.63958275	0.208333
		2	6.1516666	6.227083625	0.075417

	CICO	1	13.7625025	10.6770835	-3.08542
		2	11.94791625	9.87916575	-2.06875
	CICO+DI (update 25k)	1	38.09166625	31.13749875	-6.95417
		2	33.82291125	32.53334625	-1.28957

This table show comparison results of CICO without Data Import in two MSK clusters

Scenario	Load level	Request	Response time, sec		Response time, sec		Delta
			MSK instance: kafka.m5.2xlarge		MSK instance: kafka.m7g.2xlarge
			95 perc	Average	95 perc	Average	Average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.669	0.57	0.72	0.606	0.036
		Check-out	1.152	0.96	1.241	0.969	0.009

Response time

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

Service CPU Utilization

Delta for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The most part of modules CPU utilization fluctuate under 10%.

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

MSK instance: kafka.m5.2xlarge			MSK instance: kafka.m7g.2xlarge			DELTA
Module	CPU (CICO + 25k Create)	CPU (CICO + 25k Update)	Module	CPU (CICO + 25k Create)	CPU (CICO + 25k Update)	Delta, Create	Delta, Update
mod-inventory-b	110.54	145.5	mod-inventory-b	115.21	136.94	4.67	-8.56
mod-quick-marc-b	90.64	102.38	mod-quick-marc-b	95.15	96.4	4.51	-5.98
mod-di-converter-storage-b	78.09	121.08	mod-di-converter-storage-b	81.26	100.43	3.17	-20.65
nginx-okapi	64.1	98.19	nginx-okapi	70.58	88.94	6.48	-9.25
okapi-b	39.14	58.73	okapi-b	38.89	50.55	-0.25	-8.18
mod-source-record-storage-b	28.06	44.84	mod-source-record-storage-b	31.61	39.13	3.55	-5.71
mod-users-b	23.41	20.28	mod-users-b	23.6	22.12	0.19	1.84
mod-inventory-storage-b	20.17	24.74	mod-inventory-storage-b	21.37	19.9	1.2	-4.84
mod-source-record-manager-b	18.9	19.54	mod-feesfines-b	18.28	9.11	-0.62	-10.43
mod-feesfines-b	17.74	8.11	mod-configuration-b	17.6	10.52	-0.14	2.41
mod-configuration-b	14.55	10.3	mod-source-record-manager-b	17.39	18.27	2.84	7.97
mod-dcb-b	12.3	11.91	mod-authtoken-b	17.04	13.37	4.74	1.46
mod-authtoken-b	7.67	11.87	mod-dcb-b	13.23	12.33	5.56	0.46
mod-search-b	7.32	6	mod-search-b	7.95	1.83	0.63	-4.17
mod-pubsub-b	6.35	6.8	mod-pubsub-b	6.82	6.49	0.47	-0.31
mod-entities-links-b	3.58	2.26	pub-okapi	3.56	3.64	-0.02	1.38
pub-okapi	3.42	3.4	mod-circulation-storage-b	3.35	2.7	-0.07	-0.7
mod-patron-b	2.84	2.77	mod-patron-b	2.72	2.79	-0.12	0.02
mod-circulation-storage-b	2.83	2.91	mod-entities-links-b	2.24	2.23	-0.59	-0.68
mod-data-import-b	2.04	1.65	mod-circulation-b	1.98	1.8	-0.06	0.15
mod-circulation-b	1.92	1.6	mod-data-import-b	1.76	1.88	-0.16	0.28
edge-patron-b	1.15	1.16	edge-patron-b	1.13	1.16	-0.02	0
mod-patron-blocks-b	0.99	0.81	mod-patron-blocks-b	0.97	1	-0.02	0.19
mod-users-bl-b	0.85	2.51	mod-users-bl-b	0.68	0.68	-0.17	-1.83
pub-edge	0.07	0.07	pub-edge	0.06	0.06	-0.01	-0.01

DI MARC BIB Create and Update + CICO

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

Service Memory Utilization

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

Module	Memory (kafka.m5.2xlarge)	Memory (kafka.m7g.2xlarge)	Delta
mod-dcb-b	68.81	74.37	5.56
mod-inventory-b	68.23	70.81	2.58
mod-users-b	50.17	50.37	0.2
mod-di-converter-storage-b	48.62	46.7	-1.92
mod-feesfines-b	45.56	45.44	-0.12
mod-inventory-storage-b	45.32	33.4	-11.92
mod-source-record-storage-b	44.27	55.53	11.26
okapi-b	41.85	42.5	0.65
mod-data-import-b	41.42	43.55	2.13
mod-patron-blocks-b	41.04	42.38	1.34
mod-search-b	40.55	45.58	5.03
mod-users-bl-b	39.82	45.36	5.54
mod-configuration-b	38.78	38.68	-0.1
mod-source-record-manager-b	38.45	41.91	3.46
mod-pubsub-b	36.86	35.94	-0.92
mod-quick-marc-b	31.25	42.6	11.35
mod-patron-b	31.19	30.52	-0.67
mod-entities-links-b	27.12	34.49	7.37
mod-authtoken-b	26.17	27.32	1.15
mod-circulation-b	24.17	25.15	0.98
edge-patron-b	22.77	22.38	-0.39
mod-circulation-storage-b	20.02	22.34	2.32
nginx-okapi	4.69	4.58	-0.11
pub-okapi	4.52	4.46	-0.06
pub-edge	4.46	4.41	-0.05

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

DB CPU Utilization

Average DB CPU usage for both MSK clusters during data import is 85% during create jobs and 87& during Update jobs. Check-In/Check-Out period without DI - 15%.

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

DB Connections

Average connection count for both MSK clusters during data import is about 850 connections for create and update jobs with CI/CO. And 730 connections for CI/CO without data import

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

MSK instance resource utilization

MSK resources table

		MSK instance: kafka.m5.2xlarge	MSK instance: kafka.m7g.2xlarge	Delta	%
Disk usage	Idle state
	1	0.474	0.356	-0.118
	2	0.476	0.357	-0.119
	Under load
	1	4.6120204	4.35	-0.26202
	2	4.611104	4.35	-0.2611
CPU usage	Idle state
	1	6.431249875	6.63958275	0.208333
	2	6.1516666	6.227083625	0.075417
	Under load
CICO	1	13.7625025	10.6770835	-3.08542	-22.42%
	2	11.94791625	9.87916575	-2.06875	-17.31%
CICO+DI	1	38.09166625	31.13749875	-6.95417	-18.26%
	2	33.82291125	32.53334625	-1.28957	-3.81%

Disk usage by broker

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

CPU (User) usage by broker

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

DB load

MSK instance: kafka.m5.2xlarge

Top SQL-queries:

MSK instance: kafka.m7g.2xlarge

Top SQL-queries:

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

AppendixInfrastructure

PTF -environment qcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
MSK perf-921-g2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2

Task count for modules mod-oa-b, mod-graphql set to 0 before test start.

Modules

All qcp1 modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt
Tue Jun 04 07:31:53 UTC 2024
mod-remote-storage	4	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-ncip	4	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	4	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-agreements	4	mod-agreements:7.0.0	2	1592	1488	128	0	0	0
mod-ebsconet	4	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-organizations	4	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-consortia	2	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
edge-sip2	2	edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	128	768	88	128
mod-serials-management	4	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512
mod-settings	4	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-data-import	7	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
edge-dematic	4	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-search	4	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024
mod-inn-reach	2	mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	1024	2880	512	1024
mod-tags	4	mod-tags:2.2.0	2	1024	896	128	768	88	128
edge-courses	4	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-authtoken	5	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
mod-inventory-update	4	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	4	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	4	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	4	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	4	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	4	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	4	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-licenses	4	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	4	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-password-validator	4	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-bulk-operations	4	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	4	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	4	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	5	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	4	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	4	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	5	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	4	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	4	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	5	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	8	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
pub-edge	3	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	4	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	4	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	4	edge-ncip:1.9.2	2	1024	896	128	768	88	128
edge-inn-reach	2	edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	128	768	88	128
mod-users-bl	4	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	128	768	88	128
mod-inventory-storage	4	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-invoice	5	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-user-import	4	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	5	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	4	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	4	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	4	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	4	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-calendar	4	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-source-record-storage	4	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-event-config	4	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	4	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	4	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	4	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	4	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-pubsub	4	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-circulation	4	mod-circulation:24.2.0	2	2880	2592	1536	1814	384	512
mod-di-converter-storage	4	mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128
edge-rtac	4	edge-rtac:2.7.1	2	1024	896	128	768	88	128
edge-orders	4	edge-orders:3.0.0	2	1024	896	128	768	88	128
mod-users	5	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-template-engine	4	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	4	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	4	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	4	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	5	mod-source-record-manager:3.9.0-SNAPSHOT.330	2	5600	5000	2048	3500	384	512
nginx-edge	3	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	4	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	3	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	4	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	4	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	4	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-dcb	5	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-service-interaction	4	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-data-export	13	mod-data-export:5.0.4	1	2048	`1844`	2048	0	0	0
mod-patron	4	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-oai-pmh	4	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	4	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java	4	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	4	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-data-export-spring	4	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
mod-organizations-storage	4	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-login	4	mod-login:7.11.0	2	1440	1298	1024	768	384	512
pub-okapi	3	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	4	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Methodology/Approach

Populate ptf-mobius-testing2 cluster with topics from tenant cluster
Run CICO for 2 hours
After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
Run alone Data Imports
Create new kafka cluster
Populate NEW cluster with topics from tenant cluster
Run CICO for 2 hours
After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
Run alone Data Imports
Compare resource utilization of MSK and main KPI for CICO & DI

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Data Import with Check-ins Check-outs (Quesnelia)[non-ECS] MSK instance type comparison

Summary

Test Runs

Test Results

Comparison

Response time

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

DB Connections

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

MSK instance resource utilization

Disk usage by broker

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

CPU (User) usage by broker

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

DB load

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

Appendix

Infrastructure

Methodology/Approach