Table of Contents
Overview IN PROGRESS

This document contains the results of testing workflows Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release with a new MSK instance type. The main idea is to see how the m7g series affects FOLIO performance. Compared results for main workflows with different instance types: kafka.m5.2xlarge against kafka.m7g.2xlarge.

Ticket:

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-921

Summary

Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
- Check-in / Check-out perform better in ........
- DI durations ..........
Resource utilization
- Average CPU utilization did not exceed -----
- Memory .......
- Average DB CPU usage
- Average connection count

Test Runs

...

Test #

...

Scenario

...

Test Results

This table shows results of Check-In/Check-out and Data Import create and update jobs.

The only difference between tests is MSK cluster instance type. Cluster ptf-mobius-testing2 has kafka.m5.2xlarge and cluster PERF-921 has kafka.m7g.2xlarge

...

Check-in/Check-out without DI

...

Response time, sec
MSK instance: kafka.m5.2xlarge

...

Response time, sec
MSK instance: kafka.m7g.2xlarge

...

Comparison

The following table compares test results of two MSK clusters.

This table show comparison results of CICO without Data Import in two MSK clusters

...

Response time

MSK instance: kafka.m5.2xlarge

Image Removed

Service CPU Utilization

MSK instance: kafka.m5.2xlarge

...

title	CPU utilization MSK instance: kafka.m5.2xlarge

...

Table of Contents
Overview IN PROGRESS

This document contains the results of testing workflows Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release with a new MSK instance type. The main idea is to see how the m7g series affects FOLIO performance. Compared results for main workflows with different instance types: kafka.m5.2xlarge against kafka.m7g.2xlarge.

Ticket:

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-921

Summary

Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
- Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.
Resource utilization
- Memory utilization didn't differ a lot between two MSK clusters
- Disk consume less resources in both (idle and CICO+DI) scenarios with m7g instance type. CPU utilization 6 times bigger in idle state but 20% less under load with m7g instance type.
- Average DB CPU usage
- Average connection count

Test Runs

Test #	MSK instance type	Scenario	Load level
1	kafka.m5.2xlarge	CICO + DI MARC Bib Create	8 users + 5K, 25K sequentially
2		DI MARC Bib Create	5K, 25K sequentially
3		CICO + DI MARC Bib Update	8 users + 5K, 25K sequentially
4		DI MARC Bib Update	5K, 25K sequentially
5	kafka.m7g.2xlarge	CICO + DI MARC Bib Create	8 users + 5K, 25K sequentially
6		DI MARC Bib Create	5K, 25K sequentially
7		CICO + DI MARC Bib Update	8 users + 5K, 25K sequentially
8		DI MARC Bib Update	5K, 25K sequentially

Test Results

This table shows results of Check-In/Check-out and Data Import create and update jobs.

The only difference between tests is MSK cluster instance type. Cluster ptf-mobius-testing2 has kafka.m5.2xlarge and cluster PERF-921 has kafka.m7g.2xlarge

MSK instance: kafka.m5.2xlarge	File size	DI Duration without CI/CO	DI Duration	CI Average sec	CO Average sec
Create	5k	00:02:31	00:02:54	0.899	1.409
	25k	00:11:49	00:12:49	0.724	1.152
Update	5k	00:03:06	00:03:14	0.807	1.257
	25k	00:15:00	00:15:30	0.784	1.275
MSK instance: kafka.m7g.2xlarge
Create	5k	00:03:05	00:02:39	0.707	1.104
	25k	00:12:03	00:12:08	0.718	1.129
Update	5k	00:03:36	00:03:34	0.742	1.124
	25k	00:17:05	00:17:33	0.756	1.148

Check-in/Check-out without DI

Scenario	Load level	Request	Response time, sec MSK instance: kafka.m5.2xlarge		Response time, sec MSK instance: kafka.m7g.2xlarge
Scenario	Load level	Request	95 perc	average	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.669	0.570	0.720	0.606
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1.152	0.960	1.241	0.969

Comparison

Data Import durations and Check-In/Check-Out response time comparison

Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.

	File size	DELTA, DI	DELTA, CICO+DI	DELTA, CI	DELTA, CO
Create	5k	00:00:34	00:00:15	0.192	0.305
	25k	00:00:14	00:00:41	0.006	0.023
Update	5k	00:00:31	00:00:20	0.065	0.133
	25k	00:02:06	00:02:03	0.028	0.127

Kafka resource utilization comparison table

Disk consume less resources in both (idle and CICO+DI) scenarios with m7g instance type. CPU utilization 6 times bigger in idle state but 20% less under load with m7g instance type.

Utilized resources	Scenario/Load	Brokers	MSK instance: kafka.m5.2xlarge	MSK instance: kafka.m7g.2xlarge	Delta, %
Disk usage	Idle state
		1	0.2095713	0.026	-0.18357
		2	0.2086522	0.026	-0.18265
	CICO+DI
		1	4.6120204	4.35	-0.26202
		2	4.611104	4.35	-0.2611
CPU usage	Idle state
		1	0.93541725	6.63958275	5.704166
		2	0.78333345	6.227083625	5.44375

	CICO	1	13.7625025	10.6770835	-3.08542
		2	11.94791625	9.87916575	-2.06875
	CICO+DI	1	38.09166625	31.13749875	-6.95417
		2	33.82291125	32.53334625	-1.28957

This table show comparison results of CICO without Data Import in two MSK clusters

Scenario	Load level	Request	Response time, sec		Response time, sec		Delta
			MSK instance: kafka.m5.2xlarge		MSK instance: kafka.m7g.2xlarge
			95 perc	Average	95 perc	Average	Average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.669	0.57	0.72	0.606	0.036
		Check-out	1.152	0.96	1.241	0.969	0.009

Response time

MSK instance: kafka.m5.2xlarge

Image Added

Service CPU Utilization

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

Expand

title	CPU utilization MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m5.2xlarge			MSK instance: kafka.m7g.2xlarge			DELTA
Module	CPU (CICO + 25k Create)	CPU (CICO + 25k Update)	Module	CPU (CICO + 25k Create)	CPU (CICO + 25k Update)	Delta, Create	Delta, Update
mod-inventory-b	110.54	145.5	mod-inventory-b	115.21	136.94	4.67	-8.56
mod-quick-marc-b	90.64	102.38	mod-quick-marc-b	95.15	96.4	4.51	-5.98
mod-di-converter-storage-b	78.09	121.08	mod-di-converter-storage-b	81.26	100.43	3.17	-20.65
nginx-okapi	64.1	98.19	nginx-okapi	70.58	88.94	6.48	-9.25
okapi-b	39.14	58.73	okapi-b	38.89	50.55	-0.25	-8.18
mod-source-record-storage-b	28.06	44.84	mod-source-record-storage-b	31.61	39.13	3.55	-5.71
mod-users-b	23.41	20.28	mod-users-b	23.6	22.12	0.19	1.84
mod-inventory-storage-b	20.17	24.74	mod-inventory-storage-b	21.37	19.9	1.2	-4.84
mod-source-record-manager-b	18.9	19.54	mod-feesfines-b	18.28	9.11	-0.62	-10.43
mod-feesfines-b	17.74	8.11	mod-configuration-b	17.6	10.52	-0.14	2.41
mod-configuration-b	14.55	10.3	mod-source-record-manager-b	17.39	18.27	2.84	7.97
mod-dcb-b	12.3	11.91	mod-authtoken-b	17.04	13.37	4.74	1.46
mod-authtoken-b	7.67	11.87	mod-dcb-b	13.23	12.33	5.56	0.46
mod-search-b	7.32	6	mod-search-b	7.95	1.83	0.63	-4.17
mod-pubsub-b	6.35	6.8	mod-pubsub-b	6.82	6.49	0.47	-0.31
mod-entities-links-b	3.58	2.26	pub-okapi	3.56	3.64	-0.02	1.38
pub-okapi	3.42	3.4	mod-circulation-storage-b	3.35	2.7	-0.07	-0.7
mod-patron-b	2.84	2.77	mod-patron-b	2.72	2.79	-0.12	0.02
mod-circulation-storage-b	2.83	2.91	mod-entities-links-b	2.24	2.23	-0.59	-0.68
mod-data-import-b	2.04	1.65	mod-circulation-b	1.98	1.8	-0.06	0.15
mod-circulation-b	1.92	1.6	mod-data-import-b	1.76	1.88	-0.16	0.28
edge-patron-b	1.15	1.16	edge-patron-b	1.13	1.16	-0.02	0
mod-patron-blocks-b	0.99	0.81	mod-patron-blocks-b	0.97	1	-0.02	0.19
mod-users-bl-b	0.85	2.51	mod-users-bl-b	0.852.5168	0.68	-0.17	-1.83
pub-edge	0.07	0.07	pub-edge	0.06	0.06	-0.01	-0.01

DI MARC BIB Create and Update + CICO

...

Service Memory Utilization

MSK instance: kafka.m5.2xlarge

Expand

title	Memory consumption

Module	Memory
mod-dcb-b	68.81
mod-inventory-b	68.23
mod-users-b	50.17
mod-di-converter-storage-b	48.62
mod-feesfines-b	45.56
mod-inventory-storage-b	45.32
mod-source-record-storage-b	44.27
okapi-b	41.85
mod-data-import-b	41.42
mod-patron-blocks-b	41.04
mod-search-b	40.55
mod-users-bl-b	39.82
mod-configuration-b	38.78
mod-source-record-manager-b	38.45
mod-pubsub-b	36.86
mod-quick-marc-b	31.25
mod-patron-b	31.19
mod-entities-links-b	27.12
mod-authtoken-b	26.17
mod-circulation-b	24.17
edge-patron-b	22.77
mod-circulation-storage-b	20.02
nginx-okapi	4.69
pub-okapi	4.52
pub-edge	4.46

DB CPU Utilization

MSK instance: kafka.m5.2xlarge

Average DB CPU usage during data import is 85% during create jobs and 87& during Update jobs. Check-In/Check-Out period without DI - 15%.

DB Connections

MSK instance: kafka.m5.2xlarge

Average connection count during data import is about 850 connections for create and update jobs with CI/CO. And 730 connections for CI/CO without data import

Expand

title	MSK resources table

MSK instance: kafka.m5.2xlarge
Disk usage	Idle state
	1	0.2095713
	2	0.2086522
	Under load
	1	4.6120204
	2	4.611104
CPU usage	Idle state
	1	0.93541725
	2	0.78333345
	Under load
CICO	1	13.7625025
	2	11.9479163
CICO+DI	1	38.0916663
	2	33.8229113

Disk usage by broker

MSK instance: kafka.m5.2xlarge

Image Modified

CPU (User) usage by broker

MSK instance: kafka.m5.2xlarge

Image Modified

DB load

MSK instance: kafka.m5.2xlarge

Top SQL-queries:

AppendixInfrastructure

PTF -environment qcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Expand

title	All qcp1 modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt
Tue Jun 04 07:31:53 UTC 2024
mod-remote-storage	4	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-ncip	4	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	4	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-agreements	4	mod-agreements:7.0.0	2	1592	1488	128	0	0	0
mod-ebsconet	4	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-organizations	4	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-consortia	2	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
edge-sip2	2	edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	128	768	88	128
mod-serials-management	4	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512
mod-settings	4	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-data-import	7	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
edge-dematic	4	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-search	4	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024
mod-inn-reach	2	mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	1024	2880	512	1024
mod-tags	4	mod-tags:2.2.0	2	1024	896	128	768	88	128
edge-courses	4	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-authtoken	5	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
mod-inventory-update	4	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	4	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	4	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	4	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	4	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	4	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	4	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-licenses	4	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	4	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-password-validator	4	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-bulk-operations	4	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	4	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	4	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	5	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	4	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	4	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	5	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	4	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	4	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	5	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	8	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
pub-edge	3	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	4	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	4	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	4	edge-ncip:1.9.2	2	1024	896	128	768	88	128
edge-inn-reach	2	edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	128	768	88	128
mod-users-bl	4	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	128	768	88	128
mod-inventory-storage	4	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-invoice	5	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-user-import	4	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	5	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	4	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	4	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	4	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	4	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-calendar	4	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-source-record-storage	4	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-event-config	4	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	4	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	4	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	4	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	4	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-pubsub	4	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-circulation	4	mod-circulation:24.2.0	2	2880	2592	1536	1814	384	512
mod-di-converter-storage	4	mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128
edge-rtac	4	edge-rtac:2.7.1	2	1024	896	128	768	88	128
edge-orders	4	edge-orders:3.0.0	2	1024	896	128	768	88	128
mod-users	5	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-template-engine	4	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	4	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	4	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	4	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	5	mod-source-record-manager:3.9.0-SNAPSHOT.330	2	5600	5000	2048	3500	384	512
nginx-edge	3	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	4	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	3	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	4	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	4	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	4	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-dcb	5	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-service-interaction	4	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-data-export	13	mod-data-export:5.0.4	1	2048	`1844`	2048	0	0	0
mod-patron	4	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-oai-pmh	4	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	4	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java	4	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	4	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-data-export-spring	4	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
mod-organizations-storage	4	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-login	4	mod-login:7.11.0	2	1440	1298	1024	768	384	512
pub-okapi	3	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	4	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

...

Version	Old Version 2	New Version 3
Changes made by	Leonid Kolesnykov	Leonid Kolesnykov
Saved on	Jun 07, 2024	Jun 11, 2024

Versions Compared

Key

Table of Contents
Overview IN PROGRESS

Summary

Test Runs

Test Results

Comparison

Response time

Service CPU Utilization

MSK instance: kafka.m5.2xlarge

Table of Contents
Overview IN PROGRESS

Summary

Test Runs

Test Results

Comparison

Response time

Service CPU Utilization

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

DI MARC BIB Create and Update + CICO

Service Memory Utilization

MSK instance: kafka.m5.2xlarge

DB CPU Utilization

MSK instance: kafka.m5.2xlarge

DB Connections

MSK instance: kafka.m5.2xlarge

DB load

MSK instance: kafka.m5.2xlarge

Appendix

Infrastructure

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Page Comparison

Versions Compared

Key

Summary

Test Runs

Test Results

Comparison

Response time

Service CPU Utilization

MSK instance: kafka.m5.2xlarge

Summary

Test Runs

Test Results

Comparison

Response time

Service CPU Utilization

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

DI MARC BIB Create and Update + CICO

Service Memory Utilization

MSK instance: kafka.m5.2xlarge

DB CPU Utilization

MSK instance: kafka.m5.2xlarge

DB Connections

MSK instance: kafka.m5.2xlarge

Disk usage by broker

MSK instance: kafka.m5.2xlarge

CPU (User) usage by broker

MSK instance: kafka.m5.2xlarge

DB load

MSK instance: kafka.m5.2xlarge

Appendix

Infrastructure