Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.

PERF-702 - Getting issue details... STATUS

Summary

Check-In/Check-Out tests on Poppy carried out with 8, 20, 30, 75 users during 45 minutes and with 30 users during 14 hours - longevity test .

45 minute tests

Within Poppy release 45 min tests best response times results were with 20 users. Average CI - 0.45 sec, CO - 0.805 sec. During the highest load (75 users) response times degraded ~ 30% in average.
Comparing Poppy and Orchid tests with 20 users during 45 minutes there's no significant difference but for tests with higher vUsers amount we see improvements for CI and CO which should be investigated.

Longevity test

Within Poppy release Longevity test with 30 vUsers degraded compared to 45 min test with: CI - 0.471 sec(+4.90%), CO - 1.025 sec(+27.01%). But CI/CO is stable still during a long fixed load.
Comparing Longevity results of Orchid vs Poppy - CI response times changed from 1.061 sec to 0.471 sec and CO - from 3.233 sec to 1.025 sec.
Database utilization has growing trend for longevity test.

Common notes

Main API call slightly degraded (POST CO: +5.07%)
Memory consumption spikes at the beginning of testing usually go after cluster start and stabilize. But here memory leak for mod-pubsub module was suspected.
Spikes observed during all tests during 5-10 seconds for all requests with every minute frequency. Response times are up to 6 seconds.

Recommendations

Investigate mod-pubsub module behaviour PERF-721 - Getting issue details... STATUS
Investigate response times differences between releases. Run additional test on Orchid release with 30 vUsers for 45 minutes and compare with Poppy results to find the source of changes.

Test Runs

The following table contains information about a test model and related Grafana snapshots:

Test #	Test Conditions	vUsers	Ramp-up, sec	Duration, sec	Grafana
1.	circulation_checkInCheckOut_Poppy_3	8	80	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698852592867&to=1698855933000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
2.		20	200	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698856470216&to=1698859223000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
3.		30	300	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698861207442&to=1698864080572&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
4.		75	750	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698865254355&to=1698868102672&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
5. Retest		75	750	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698915649265&to=1698918502961&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
6.		30	300	86400	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1699029000000&to=1699087095584&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Results

The following table contains information about test results and their comparison with the baseline (poppy 8 VU).

Comparing Longevity test results against 45 min with 30 users within Poppy release we see almost the same results for Check-In 0.471 sec(+4.90%) in average and 1.025 sec(+27.01%) for Check-Out.
Within Poppy the response times during test with 75 users grew up to 30% compared to 8 users test.

Errors:

Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. Reason for that is a doubled usage of checked-out item barcodes.

Response time

Controller \ Users	8 users
	Requests	75th pct	95th pct	Average
Check-In (baseline)	1400	0.445	0.599	0.479
Check-Out (baseline)	1807	0.814	1.016	0.832
	20 users
Check-In	3439	0.426 (-4.27%)	0.503 (-16.03%)	0.45 (-6.05%)
Check-Out	4432	0.785 (-3.56%)	0.931 (-8.37%)	0.805 (-3.25%)
	30 users
Check-In	4873	0.422 (-5.17%)	0.488 (-18.53%)	0.449 (-6.26%)
Check-Out	6759	0.786 (-3.44%)	0.912 (-10.24%)	0.807 (-3%)
	75 users With nginx okapi issues
Check-In	10953	0.592 (+33.03%)	0.733 (+22.37%)	0.57 (+19%)
Check-Out	14532	1.137 (+39.68%)	1.378 (+35.63%)	1.059 (+27.28%)
	75 users Retest
Check-In	11046	0.641 (+44.04%)	0.785 (+31.05%)	0.622 (+29.85%)
Check-Out	14478	1.182 (+45.21%)	1.453 (+43.01%)	1.116 (+34.13%)
	30 users Longevity test (compared to 30 vUsers)
Check-In	110679	0.444 (+5.21%)	0.504 (+3.28%)	0.471 (+4.90%)
Check-Out	147181	1.068 (+35.88%)	1.195 (+31.03%)	1.025 (+27.01%)

This table contains common information about tests, amount of total and failed requests, average requests per second, 95th percentile of response time..

Sampler Type	8 users	20 users	30 users	75 users with fails	75 users Retest	30 users Longevity
	Value	Value	Value	Value	Value	Value
Total Requests	113765	276089	415857	908767	907566	9170318
Failed Requests	1	2	1	1410	16	198
Average RPS	42	104	154	337	337	158
Min, ms	0	0	0	0	0	0
Median, ms	7	7	7	9	9	7
Percentile 95, ms	176	167	164	239	251	177

Comparisons

The table contains comparison results for Poppy and Orchid releases:

Test #	Test Conditions	vUsers	Action	95th pct	Average
	circulation_checkInCheckOut_orchid (baseline)	8	Check-in	1.027	0.534
			Check-out	1.201	0.811
	circulation_checkInCheckOut_Poppy_3	8	Check-in	0.599 (-41.67%)	0.479 (-10.30%)
			Check-out	1.016 (-15.40%)	0.832 (+2.59%)
	circulation_checkInCheckOut_orchid (baseline)	20	Check-in	0.812	0.513
			Check-out	1.130	0.798
	circulation_checkInCheckOut_Poppy_3	20	Check-in	0.503 (-38.05%)	0.45 (-12.28%)
			Check-out	0.931 (-17.61%)	0.805 (+0.88%)
	circulation_checkInCheckOut_orchid (baseline)	30	Check-in	1.708	0.834
			Check-out	2.77	1.59
	circulation_checkInCheckOut_Poppy_3	30	Check-in	0.488 (-71.43%)	0.449 (-46.16%)
			Check-out	0.912 (-67.08%)	0.807 (-49.25%)
	circulation_checkInCheckOut_orchid (baseline)	75	Check-in	1.566	0.825
			Check-out	3.142	1.960
	circulation_checkInCheckOut_Poppy_3	75	Check-in	0.785 (-49.87%)	0.622 (-24.61%)
			Check-out	1.453 (-53.76%)	1.116 (-43.06%)
	circulation_checkInCheckOut_orchid (baseline) Longevity	30	Check-in	0.550	1.061
			Check-out	1.597	3.233
	circulation_checkInCheckOut_Poppy_3 Longevity	30	Check-in	0.504 (-8.36%)	0.471 (-55.61%)
			Check-out	1.195 (-25.17%)	1.025 (-68.30%)

The following table contains information about the response time of the most important requests for CI/CO activities and their comparison with Orchid's results:

API	8 users Orchid 75pct, ms	8 users Poppy 75pct, ms	20 users Orchid 75pct, ms	20 users Orchid 75pct, ms
POST checkout-by-barcode	276	290 +5.07%	264	281 +6.44%
POST checkin-by-barcode	258	181 -29.84%	245	174 -28.98%
GET circulation/loans	140	139 -0.71%	136	132 -2.94%

API

8 users Orchid

75pct, ms

8 users Poppy

75pct, ms

20 users Orchid

75pct, ms

20 users Orchid

75pct, ms

POST checkout-by-barcode

276

290

+5.07%

264

281

+6.44%

POST checkin-by-barcode

258

181

-29.84%

245

174

-28.98%

GET circulation/loans

140

139

-0.71%

136

132

-2.94%

Resources Utilization

Test	VU	Memory Consumption	CPU Utilization	RDS CPU Utilization / CPU (User) usage by broker
1	8			CPU (User) usage by broker didn't exceed 23.3%
2	20
3	30
4	75 failed
5	75 retest	On the first graph slow growing for mod-pubsub service was detected. And this trend clearly seen in longevity test results.	nginx-okapi go up to 800% after 10 minutes of test start. This graph doesn't contain nginx-okapi. The highest cpu utilization from mod-users module - 111%. okapi-b uses goes with 80%, the rest of modules doesn't exceed 65%.	RDS CPU utilization average was 60% and didn't exceed 70% in spikes. CPU (User) usage by broker didn't exceed 40%
6	30	Here we can see that mod-pubsub grew up to 53%.		CPU (User) usage by broker didn't exceed 23% and at 22:10 UTC went down to 13%

Appendix

Infrastructure

PTF -environment ncp5	PTF -environment pcp1
8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region] 2 instances of db.r6.xlarge database instances: Writer & reader instances MSK ptf-kakfa-3 [ kafka configurations] 4 kafka.m5.2xlarge brokers in 2 zones Apache Kafka version 2.8.0 EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=3	10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 2 database instances db.r6g.xlarge, writer/reader, Memory GIB-32GiB, vCPUs - 4 vCPUs, max. connections: 2731 MSK tenant 4 m5.2xlarge brokers in 2 zones Apache Kafka version 2.8.0 EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=3

PTF -environment ncp5

PTF -environment pcp1

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances db.r6g.xlarge, writer/reader, Memory GIB-32GiB, vCPUs - 4 vCPUs, max. connections: 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcp1-pvt

mod-remote-storage	10	mod-remote-storage:3.0.0	2	4920	4472	1024	3960	512	512	FALSE
mod-authtoken	9	mod-authtoken:2.14.0	2	1440	1152	512	922	88	128	FALSE
mod-configuration	9	mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	9	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	10	mod-inventory-storage:27.0.0	2	4096	3690	2048	3076	384	512	FALSE
mod-circulation-storage	10	mod-circulation-storage:17.1.0	2	2880	2592	1536	1814	384	512	FALSE
mod-inventory	9	mod-inventory:20.1.0	2	2880	2592	1024	1814	384	512	FALSE
mod-circulation	10	mod-circulation:24.0.0	2	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	9	mod-pubsub:2.11.0	2	1536	1440	1024	922	384	512	FALSE
mod-patron-blocks	9	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	FALSE
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	10	okapi:5.0.1	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	10	mod-feesfines:19.0.0	2	1024	896	128	768	88	128	FALSE

Methodology/Approach

Run data preparation script before each CI/CO test
Update .jmx file script for Poppy release
Create artefact and upload to carrier-io
Use Jenkins job to change parameters and run tests
Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
Test CI/CO with 30 users for 24 hours to detect any trends in memory.

Overview

Summary

Recommendations

Test Runs

Results

Comparisons

CPU (User) usage by broker didn't exceed 23.3%

CPU (User) usage by broker didn't exceed 40%

CPU (User) usage by broker didn't exceed 23% and at 22:10 UTC went down to 13%

Infrastructure

Methodology/Approach

Browser not supported