Overview

Tests were conducted in scope of PERF-597 task to assess performance of the Collection Service functionality (RIH scenario in JMeter script) and get the baseline numbers for future testing.

Summary

There is no difference in performance of functionality between normal and high load tests.
There is response time degradation in comparison with tests for all the workflows running at the same time. Details can be found at the Test results part.
During load test CPU utilization for nginx-okapi, mod-inventory-storage, pub-okapi, okapi-b and mod-source-record-storage modules increased to max 5%. Rest of the modules CPU utilization didn't change.
Service memory utilization was stable, and no memory leaks were suspected during tests.

Test Runs

Test #	Test type	# configuration	Test duration	Comment
1	Normal load	RIH scenario with load of 3 requests/second	1 hour	Tests were conducted 2 times each, no differences in results were observed
2	High load	RIH scenario with load of 20 requests/second	1 hour

Test Results

Test results from 1st test run (1st and 2nd test run results are similar):

Request name	Response time normal load						Response time high load
	Individual workflow		All the workflows*		All the workflows + DI + FY*		Individual workflow		All the workflows*		All the workflows + FY*
	AVG (sec)	95 PCTL (sec)	AVG (sec)	95 PCTL (sec)	AVG (sec)	95 PCTL (sec)	AVG (sec)	95 PCTL (sec)	AVG (sec)	95 PCTL (sec)	AVG (sec)	95 PCTL (sec)
RIH_1GET source-storage/source-records	0.019	0.024	0.020	0.038 (+58%)	0.031	0.066 (+175%)	0.019	0.024	0.023	0.043 (+79%)	0.030	0.047 (+95%)
RIH_2GET holdings-storage/holdings	0.015	0.017	0.015	0.033 (+94%)	19.799**	30.018**(+30 sec)	0.014	0.017	0.019	0.041 (+141%)	0.027	0.049 (+188%)

*previous test results can be found here: Master Script normal load test - NLA report and Master Script high load test - NLA report (without Data Import)

**error rate for this request during the test was 35%

Instance CPU Utilization

normal load

Instance CPU Utilization did not exceed 7,5%.

high load

Instance CPU Utilization did not exceed 8,5%, except 1 spike at the beginning of the test.

Service CPU Utilization

normal load

During load test CPU utilization for nginx-okapi, mod-inventory-storage, pub-okapi and okapi-b modules increased to 1-1,5%. Rest of the modules CPU didn't change.

high load

During load test CPU utilization for nginx-okapi, mod-inventory-storage, pub-okapi and okapi-b modules increased to 5%, CPU utilization for mod-source-record-storage increased to 2%. Rest of the modules CPU didn't change.

Service Memory Utilization

normal load

This graph represents memory usage of test runs and shows that no memory leak is suspected for all of the modules.

high load

This graph represents memory usage of test runs and shows that no memory leak is suspected for all of the modules.

DB CPU Utilization

normal load

DB CPU usage was at the same level as without load - about 7,5%.

high load

DB CPU usage increased from 7,2% to 8,5%.

DB Connections

normal load

Connections count didn't change during the test.

high load

Connections count didn't change during the test.

Appendix

Infrastructure

PTF -environment ncp3

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-inventory-storage	26.0.0	1	2	1024	2208	1952	384	1440
mod-inventory	20.0.4	1	2	1024	2880	2592	512	1814
mod-tags	2.0.1	1	2	128	1024	896	128	768
mod-gobi	2.6.0	1	2	128	1024	896	128	700
mod-remote-storage	2.0.2	1	2	1024	4920	4472	512	3960
mod-invoice-storage	5.6.0	1	2	128	1024	896	128	700
edge-sip2	3.0.0	1	2	128	1024	896	128	768
mod-users-bl	7.5.0	1	2	512	1440	1152	128	922
edge-rtac	2.6.0	1	2	128	1024	896	128	768
mod-feesfines	18.2.1	1	2	128	1024	896	128	768
mod-rtac	3.5.0	1	2	128	1024	896	128	768
mod-erm-usage-harvester	4.3.0	1	2	128	1024	896	128	768
mod-search	2.0.1	1	2	400	2592	2480	1024	1440
mod-service-interaction	2.2.2	1	2	256	2048	1844	512	1290
edge-ncip	1.8.1	1	2	128	1024	896	128	768
mod-authtoken	2.13.0	1	2	512	1440	1152	128	922
mod-permissions	6.3.1	2	2	512	1684	1544	512	1024
mod-circulation-storage	16.0.0	1	2	1024	1536	1440	512	896
mod-ncip	1.13.1	1	2	128	1024	896	128	768
mod-pubsub	2.9.1	1	2	1024	1536	1440	512	922
edge-orders	2.8.1	1	2	1024	1536	1440	512	922
mod-circulation	23.5.4	1	2	1536	2880	2592	128	700
edge-caiasoft	2.0.0	1	2	128	1024	896	-	-
mod-data-export	4.7.1	1	1	1024	1024	896	128	768
mod-organizations-storage	4.5.1	1	2	128	1024	896	128	700
mod-source-record-storage	5.6.5	1	2	2048	5600	5000	512	3600
mod-copycat	1.4.0	1	2	896	1024	896	128	768
mod-bulk-operations	1.0.5	1	2	1024	3072	2600	512	1536
mod-quick-marc	3.0.0	1	1	128	2288	2176	512	1664
mod-audit	2.7.0	1	2	1024	1024	896	128	768
mod-oai-pmh	3.11.3	1	2	1024	2248	2000	512	1440
edge-connexion	1.0.6	1	2	128	1024	896	128	768
mod-kb-ebsco-java	3.13.0	1	2	128	1024	896	128	768
mod-patron	5.5.2	1	2	128	1024	896	128	768
mod-email	1.15.3	1	2	128	1024	896	128	768
mod-password-validator	3.0.0	1	2	128	1440	1298	512	768
mod-login	7.9.0	1	2	1024	1440	1298	512	768
mod-data-export-worker	3.0.12	1	2	1024	3072	2600	512	2048
mod-agreements	5.5.2	1	2	128	3096	2580	512	2048
edge-oai-pmh	2.6.1	1	2	1024	1512	1360	512	1440
mod-eusage-reports	1.3.0	1	2	128	1024	896	128	768
mod-orders-storage	13.5.0	1	2	512	1024	896	128	700
mod-notify	3.0.0	1	2	128	1024	896	128	768
mod-source-record-manager	3.6.2	1	2	2048	5600	5000	512	3600
mod-di-converter-storage	2.0.2	2	2	128	1024	896	128	768
mod-template-engine	1.18.0	1	2	128	1024	896	128	768
mod-user-import	3.7.2	1	2	128	1024	896	128	768
mod-finance-storage	8.4.1	1	2	128	1024	896	128	700
mod-users	19.1.1	1	2	128	1024	896	128	768
mod-sender	1.10.0	1	2	128	1024	896	128	768
mod-graphql	1.11.0	1	2	128	1024	896	128	768
mod-licenses	4.3.1	1	2	128	2480	2312	512	1792
mod-invoice-b	5.6.2	1	2	512	1440	1152	128	922
mod-event-config	2.5.0	1	2	128	1024	896	128	768
mod-calendar	2.4.2	1	2	128	1024	896	128	768
mod-erm-usage	4.5.2	1	2	128	1024	896	128	768
mod-patron-blocks	1.8.0	1	2	1024	1024	896	128	768
mod-data-import	2.7.1	1	1	256	2048	1844	512	1292
mod-ebsconet	2.0.0	1	2	128	1248	1024	256	700
edge-dematic	2.0.0	1	2	128	1024	896	-	-
mod-task-list	5.0.1	1	1	128	1024	896	128	768
mod-courses	1.4.7	1	2	128	1024	896	128	768
mod-inventory-update	3.0.1	1	2	128	1024	896	128	768
mod-login-saml	2.6.1	1	2	128	1024	896	128	768
mod-orders	12.6.6	1	2	1024	2048	1440 (Recommended to change to 1544)	512	1024
mod-configuration	5.9.1	1	2	128	1024	896	128	768
mod-organizations	1.7.0	1	2	128	1024	896	128	700
mod-notes	5.0.1	1	2	128	1024	896	128	322
mod-finance	4.7.1	1	2	128	1024	896	128	700
mod-data-export-spring	2.0.1	1	1	256	2048	1844	256	1292
edge-patron	4.11.0	1	2	256	1024	896	128	768
okapi	5.0.1	2	3	1024	1684	1440	512	922
nginx-okapi	2022.03.02	1	2	128	1024	896	-	-
pub-okapi	2022.03.02	1	2	128	1024	896	-	768

Methodology/Approach

To assess performance of the Collection Service functionality and get the baseline numbers the JMeter script was used (RIH scenario).

Additional Screenshots of graphs or charts

Previous test results:

Master Script normal load test - NLA report

Master Script high load test - NLA report (without Data Import)

Grafana dashboard:

normal load: http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1686038285285&to=1686042157683&var-percentile=95&var-test_type=baseline&var-test=master-script-longevity&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

high load: http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1686051528312&to=1686055298858&var-percentile=95&var-test_type=baseline&var-test=master-script-longevity&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Folio Development Teams

Retrieving instances and holdings - NLA report

Overview

Summary

Test Runs

Test Results

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

Appendix

Infrastructure

Methodology/Approach

Additional Screenshots of graphs or charts

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731