Overview

This document contains the results of testing different configurations of modules with changes in JAVA_OPTS replacing XMX with MaxRAMPercentage. The main KPI for these tests are Memory consumption of modules, Data import durations and Check-in / Check-out response times.

Ticket: PERF-893 - Getting issue details... STATUS

Summary

Replacing XMX with MaxRAMPercentage do not affect main KPI's. Comparing DI's durations in tests #1 and #2 deltas do not exceed 4 minutes that is acceptable.
After changing MaxRAMPercentage from default values to recommended 66% Check-in / Check-out response times increased during data import for CI - 15%, for CO - 7%, and it also affected Data import durations - 8 minutes difference in DI test #6.
Comparing DI durations between configuration #2 and #3 (with limited MaxRAMPercentage) we see that DI slower in configuration #3. Previously known issue with mod-inventory (container stop because of CPU growing) appear much faster in configuration #3 with limited MaxRAMPercentage.

CPU utilization top three modules:

Service	Configuration #1	Configuration #2	Configuration #3
mod-inventory-b	254.27	305.56	216.98
mod-quick-marc-b	120.83	58.49	87.44
nginx-okapi	91.73	56.1	95.49

Memory consumption top three modules:

Service	Configuration #1	Configuration #2	Configuration #3
mod-inventory-b	99.4	105.84	102.32
mod-search-b	96.43	89.1	96.81
mod-users-b	63.08	61.58	60.63

Service CPU utilization changed between tests because of the cluster restarts
RDS CPU utilization was close to 96% during CI/CO + Data import
RDS DB Connections were close to 600 during CI/CO + Data import

Test Runs

Test / Configuration #	Scenario	Load level	`JAVA_OPTS`
1	20 vUsers longevity CI/CO + DI MARC Bib Create	12 Data Import tests with 100K sequentially (with 5 min pause)	`-XMX`
2	20 vUsers longevity CI/CO + DI MARC Bib Create	12 Data Import tests with 100K sequentially (with 5 min pause)	`-XX:MaxRAMPercentage, Default`
3	20 vUsers longevity CI/CO + DI MARC Bib Create	12 Data Import tests with 100K sequentially (with 5 min pause)	`-XX:MaxRAMPercentage, Recommended - 66%`

Test Results

Comparison

The following table compares test results

Configuration #1 BASELINE, XMX			Configuration #2 default MaxRAMPercentage				Configuration #3 recommended MaxRAMPercentage, 66%
16 hours CICO	Test #	hh:mm:ss	24 hours CI/CO	Test #	hh:mm:ss	delta, configuration #2 - #1	24 hours CI/CO	Test #	hh:mm:ss	delta, configuration #3 - #1
								DI 100k Create #12	00:56:32	These results after 1 container for mod-inventory stopped. A new 1 container was created.
				DI 100k Create #11	01:39:09	4 1k jobs stopped		DI 100k Create #11	00:55:58
				DI 100k Create #10	01:03:19			DI 100k Create #10	00:57:00
				DI 100k Create #9	00:59:40			DI 100k Create #9	00:56:10
	DI 100k Create #8	00:55:01		DI 100k Create #8	00:58:47	00:03:46		DI 100k Create #8	00:55:23
	DI 100k Create #7	00:54:43		DI 100k Create #7	00:58:08	00:03:26		DI 100k Create #7	02:46:27	3 1k jobs stopped
	DI 100k Create #6	00:55:25		DI 100k Create #6	00:56:09	00:00:44		DI 100k Create #6	01:03:07	00:07:42
	DI 100k Create #5	00:54:45		DI 100k Create #5	00:58:19	00:03:33		DI 100k Create #5	01:03:37	00:08:52
	DI 100k Create #4	00:55:38		DI 100k Create #4	00:54:27	00:01:11		DI 100k Create #4	00:59:11	00:03:33
	DI 100k Create #3	00:53:30		DI 100k Create #3	00:54:36	00:01:06		DI 100k Create #3	00:58:34	00:05:04
	DI 100k Create #2	00:53:16		DI 100k Create #2	00:53:03	00:00:13		DI 100k Create #2	00:57:59	00:04:43
	DI 100k Create #1	00:52:51		DI 100k Create #1	01:05:10	00:12:18		DI 100k Create #1	01:05:42	00:12:50

Results from test #8	DI +	DI -		DI +	DI -			DI +	DI -	Comparison CI/CO + DI between configuration #3 and #2, %
CI	1220	538	CI	1078	486		CI	1240	470	15%
CO	2311	978	CO	1967	1041		CO	2121	965	7%

Detailed CICO response time for CICO with DI

Request*	Response time (avg, sec)
Request*	Pure CICO	CICO + 100K MARC BIB Create
Check-Out Controller	0.965	2.121
Check-In Controller	0.47	1.24
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	0.314	0.879
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	0.208	0.73
GET_circulation/loans (Submit_barcode_checkout)	0.142	0.259
GET_circulation/loans (Submit_patron_barcode)	0.1	0.203
GET_inventory/items (Submit_barcode_checkin)	0.055	0.182

*Top requests were taken where response times >= 100ms.

Service CPU Utilization

CPU utilization all tests

AVERAGES
Service	Configuration #1	Configuration #2	Configuration #3
mod-inventory-b	254.27	305.56	216.98
mod-quick-marc-b	120.83	58.49	87.44
nginx-okapi	91.73	56.1	95.49
mod-di-converter-storage-b	74	33.08	80.35
okapi-b	45.69	32.3	44.08
mod-source-record-storage-b	39.4	19.14	48.36
mod-users-b	38.27	33.1	33.79
mod-source-record-manager-b	29.54	15.23	32.39
mod-inventory-storage-b	21.93	13.17	20.09
mod-authtoken-b	19.67	15.99	19.56
mod-configuration-b	17.72	16.55	14.97
mod-feesfines-b	17.12	16.71	15.97
mod-pubsub-b	10.19	9.52	9.74
mod-permissions-b	8.07	4.73	6.74
pub-okapi	7.74	8.41	6.47
mod-circulation-storage-b	3.72	3.08	3.28
mod-circulation-b	3.03	3.05	2.95
mod-password-validator-b	2.71	2.4	2.65
mod-data-import-b	1.81	2.95	1.98

Configuration #1

Configuration #2

Configuration #3

Service Memory Utilization

Memory utilization all tests

AVERAGES
Service	Configuration #1	Configuration #2	Configuration #3
mod-inventory-b	99.4	105.84	102.32
mod-search-b	96.43	89.1	96.81
mod-users-b	63.08	61.58	60.63
mod-source-record-storage-b	51.69	44.76	60.69
mod-permissions-b	46.5	47.38	47.24
okapi-b	45	44.86	50.14
mod-pubsub-b	42.57	42.43	51.67
mod-data-import-b	41.87	41.54	50.16
mod-di-converter-storage-b	38.02	39.78	39.01
MAXIMUM+B47:B65	37.1	43.43	45.35
mod-feesfines-b	33.2	33.2	33.43
mod-inventory-storage-b	30.73	46.72	37.57
mod-quick-marc-b	29.74	31.84	29.61
mod-configuration-b	28.8	28.85	29.21
mod-circulation-b	25.77	25.28	26.93
mod-authtoken-b	21.52	21.45	21.45
mod-circulation-storage-b	21.05	19.83	25.98
nginx-okapi	4.88	4.97	4.97
pub-okapi	4.58	4.58	4.58

MAXIMUM
Service	Configuration #1	Configuration #2	Configuration #3
mod-inventory-b	107.68	106.21	107.65
mod-search-b	100.22	81.1	100.16
mod-users-b	62.89	60.27	62.83
okapi-b	62.64	37.74	62.64
mod-pubsub-b	59.38	34.72	59.24
mod-source-record-manager-b	52.08	29.94	52.08
mod-permissions-b	51.36	44.69	51.17
mod-inventory-storage-b	50.08	46.19	50.16
mod-source-record-storage-b	49.54	48.02	49.54
mod-data-import-b	41.81	41.65	41.81
mod-di-converter-storage-b	39.84	38.62	39.96
mod-feesfines-b	33.48	33.11	33.48
mod-quick-marc-b	32.33	32.08	32.49
mod-configuration-b	29.24	28.46	29.25
mod-circulation-b	26.27	24.42	26.27
mod-circulation-storage-b	23.03	22.8	22.96
mod-authtoken-b	21.79	21.44	21.79
nginx-okapi	5.13	5.02	5.13
pub-okapi	4.69	4.58	4.69

Configuration #1

Configuration #2

Configuration #3

DB CPU Utilization

Configuration #1

Configuration #2

Configuration #3

DB Connections

Configuration #1

Configuration #2

Configuration #3

DB load

Configuration #1

Configuration #2

Configuration #3

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module	Task Def. Revision	Task Def. Revision with MaxRAMPercentage = by default	Task Def. Revision with MaxRAMPercentage = 66.0	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	MaxRAMPercentage Tests	MaxRAMPercentage = recommended (66%)
pcp1-pvt
Thu Apr 04 09:40:28 UTC 2024
mod-data-import	20	23	24	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	70.1	66.0
mod-authtoken	16	17	18	mod-authtoken:2.14.1	2	1440	1152	512	922	88	128	80.0	66.0
mod-configuration	10	12	13	mod-configuration:5.9.2	2	1024	896	128	768	88	128	85.7	66.0
mod-users-bl	10	11	12	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	80.0	66.0
mod-inventory-storage	15	17	18	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	83.4	66.0
mod-circulation-storage	14	15	16	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	70.0	66.0
mod-source-record-storage	18	19	20	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	70.0	66.0
mod-inventory	15	16	17	mod-inventory:20.1.8	2	2880	2592	1024	1814	384	512	70.0	66.0
mod-di-converter-storage	18	19	20	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128	85.7	66.0
mod-circulation	14	15	16	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512	70.0	66.0
mod-pubsub	13	14	15	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512	64.0	66.0
mod-users	34	35	36	mod-users:19.3.0-SNAPSHOT.677	2	1024	896	128	768	88	128	85.7	66.0
mod-patron-blocks	10	11	12	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	85.7	66.0
mod-source-record-manager	17	19	20	mod-source-record-manager:3.7.8	2	5600	5000	2048	3500	384	512	70.0	66.0
nginx-okapi	9			nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	0.0
okapi-b	11	12	13	okapi:5.1.2	3	1684	1440	1024	922	384	512	64.0	66.0
mod-feesfines	11	12	13	mod-feesfines:19.0.0	2	1024	896	128	768	88	128	85.7	66.0

Methodology/Approach

Run CI/CO for 24 hours.
Run DI script with 12 DI's sequentially after ramp-up period for CI/CO.
After getting baseline, repeat CI/CO with changed revisions of modules with needed configuration of JAVA_OPTS - replaced XMX with MaxRAMPercentage.
Before tests restart the cluster to get consistent results.

Additional Screenshots of graphs or charts

Spreadsheet with notes

Folio Development Teams

MaxRAMPercentage vs Xmx with Data Import and Check-ins Check-outs (Poppy)

Overview

Summary

Test Runs

Test Results

Comparison

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Appendix

Infrastructure

Methodology/Approach

Additional Screenshots of graphs or charts

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731