Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.

https://folio-org.atlassian.net/browse/PERF-702

Summary

Check-In/Check-Out tests on Poppy carried out with 8, 20, 30, 75 users during 45 minutes and with 30 users during 14 hours - longevity test .

45 minute tests

Within Poppy release 45 min tests best response times results were with 20 users. Average CI - 0.45 sec, CO - 0.805 sec. During the highest load (75 users) response times degraded ~ 30% in average.
Comparing Poppy and Orchid tests with 20 users during 45 minutes there's no significant difference but for tests with higher vUsers amount we see improvements for CI and CO which should be investigated.

Longevity test

Within Poppy release Longevity test with 30 vUsers degraded compared to 45 min test with: CI - 0.471 sec(+4.90%), CO - 1.025 sec(+27.01%). But CI/CO is stable still during a long fixed load.
Comparing Longevity results of Orchid vs Poppy - CI response times changed from 1.061 sec to 0.471 sec and CO - from 3.233 sec to 1.025 sec.
Database utilization has growing trend for longevity test.

Common notes

Main API call slightly degraded (POST CO: +5.07%)
Memory consumption spikes at the beginning of testing usually go after cluster start and stabilize. But here memory leak for mod-pubsub module was suspected.
Spikes observed during all tests during 5-10 seconds for all requests with every minute frequency. Response times are up to 6 seconds.

Recommendations

Investigate mod-pubsub module behaviour https://folio-org.atlassian.net/browse/PERF-721
Investigate response times differences between releases. Run additional test on Orchid release with 30 vUsers for 45 minutes and compare with Poppy results to find the source of changes.

Test Runs

The following table contains information about a test model and related Grafana snapshots:

Test #	Test Conditions	vUsers	Ramp-up, sec	Duration, sec	Grafana
1.	circulation_checkInCheckOut_Poppy_3	8	80	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698852592867&to=1698855933000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
2.		20	200	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698856470216&to=1698859223000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
3.		30	300	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698861207442&to=1698864080572&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
4.		75	750	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698865254355&to=1698868102672&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
5. Retest		75	750	2700	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1698915649265&to=1698918502961&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
6.		30	300	86400	http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1699029000000&to=1699087095584&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Results

The following table contains information about test results and their comparison with the baseline (poppy 8 VU).

Comparing Longevity test results against 45 min with 30 users within Poppy release we see almost the same results for Check-In 0.471 sec(+4.90%) in average and 1.025 sec(+27.01%) for Check-Out.
Within Poppy the response times during test with 75 users grew up to 30% compared to 8 users test.

Errors:

Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. Reason for that is a doubled usage of checked-out item barcodes.

Response time

Controller \ Users	8 users
	Requests	75th pct	95th pct	Average
Check-In (baseline)	1400	0.445	0.599	0.479
Check-Out (baseline)	1807	0.814	1.016	0.832
	20 users
Check-In	3439	0.426 (-4.27%)	0.503 (-16.03%)	0.45 (-6.05%)
Check-Out	4432	0.785 (-3.56%)	0.931 (-8.37%)	0.805 (-3.25%)
	30 users
Check-In	4873	0.422 (-5.17%)	0.488 (-18.53%)	0.449 (-6.26%)
Check-Out	6759	0.786 (-3.44%)	0.912 (-10.24%)	0.807 (-3%)
	75 users With nginx okapi issues
Check-In	10953	0.592 (+33.03%)	0.733 (+22.37%)	0.57 (+19%)
Check-Out	14532	1.137 (+39.68%)	1.378 (+35.63%)	1.059 (+27.28%)
	75 users Retest
Check-In	11046	0.641 (+44.04%)	0.785 (+31.05%)	0.622 (+29.85%)
Check-Out	14478	1.182 (+45.21%)	1.453 (+43.01%)	1.116 (+34.13%)
	30 users Longevity test (compared to 30 vUsers)
Check-In	110679	0.444 (+5.21%)	0.504 (+3.28%)	0.471 (+4.90%)
Check-Out	147181	1.068 (+35.88%)	1.195 (+31.03%)	1.025 (+27.01%)

This table contains common information about tests, amount of total and failed requests, average requests per second, 95th percentile of response time..

Sampler Type	8 users	20 users	30 users	75 users with fails	75 users Retest	30 users Longevity
	Value	Value	Value	Value	Value	Value
Total Requests	113765	276089	415857	908767	907566	9170318
Failed Requests	1	2	1	1410	16	198
Average RPS	42	104	154	337	337	158
Min, ms	0	0	0	0	0	0
Median, ms	7	7	7	9	9	7
Percentile 95, ms	176	167	164	239	251	177

Comparisons

The table contains comparison results for Poppy and Orchid releases:

Test #	Test Conditions	vUsers	Action	95th pct	Average
	circulation_checkInCheckOut_orchid (baseline)	8	Check-in	1.027	0.534
			Check-out	1.201	0.811
	circulation_checkInCheckOut_Poppy_3	8	Check-in	0.599 (-41.67%)	0.479 (-10.30%)
			Check-out	1.016 (-15.40%)	0.832 (+2.59%)
	circulation_checkInCheckOut_orchid (baseline)	20	Check-in	0.812	0.513
			Check-out	1.130	0.798
	circulation_checkInCheckOut_Poppy_3	20	Check-in	0.503 (-38.05%)	0.45 (-12.28%)
			Check-out	0.931 (-17.61%)	0.805 (+0.88%)
	circulation_checkInCheckOut_orchid (baseline)	30	Check-in	1.708	0.834
			Check-out	2.77	1.59
	circulation_checkInCheckOut_Poppy_3	30	Check-in	0.488 (-71.43%)	0.449 (-46.16%)
			Check-out	0.912 (-67.08%)	0.807 (-49.25%)
	circulation_checkInCheckOut_orchid (baseline)	75	Check-in	1.566	0.825
			Check-out	3.142	1.960
	circulation_checkInCheckOut_Poppy_3	75	Check-in	0.785 (-49.87%)	0.622 (-24.61%)
			Check-out	1.453 (-53.76%)	1.116 (-43.06%)
	circulation_checkInCheckOut_orchid (baseline) Longevity	30	Check-in	0.550	1.061
			Check-out	1.597	3.233
	circulation_checkInCheckOut_Poppy_3 Longevity	30	Check-in	0.504 (-8.36%)	0.471 (-55.61%)
			Check-out	1.195