Overview

The test aims to investigate the variability level of performance test results for CI/CO actions.

PERF-455 - Getting issue details... STATUS

Summary

Results show us that increasing amount of samples decreases the margin of error (MoE) and makes results more efficient, so a recommendation could be used following:
- increase the number of virtual users for an increasing amount of samples for the same duration
- increase the duration for a growing amount of samples
The median of the test data set is more stable than the average value of the same data set
Based on calculated results:
- for CI/CO 8VU:
  - Check-in & Check-OUT AVG +- ~6.66% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)
  - Check-in Standard Deviation of the following metrics aggregations:
    - AVG - 6.53 ms
    - Median - 5.57 ms
    - 95pct - 11.31 ms
  - Check-out Standard Deviation of the following metrics aggregations:
    - AVG - 6.81 ms
    - Median - 8.11 ms
    - 95pct - 10.39 ms
- for CI/CO 20 VU:
  - Check-in & Check-OUT AVG +- ~3.43% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)
  - Check-in Standard Deviation of the following metrics aggregations:
    - AVG - 8.07 ms
    - Median - 8.97 ms
    - 95pct - 9.79 ms
  - Check-out Standard Deviation of the following metrics aggregations:
    - AVG - 8.40 ms
    - Median - 8.70 ms
    - 95pct - 9.48 ms

Test config

Test #	Test Conditions	Duration	Count	Load generator size (recommended)	Load generator Memory(GiB) (recommended)
1.	CI/CO 8 VU	30 min	10	t3.medium	1
2.	CI/CO 20 VU	30 min	10	t3.medium	1

Results

CI/CO 8 VU

							Response time
ID	Test	Job	Total	Request group	Samples	AVG, ms	Median, ms	95pct, ms	Min, ms	Max, ms	Range, ms	StdDev, ms	MoE (95% CL)	CoV
1	CICO 8 VU	#2190	76885	Check-in	910	492	479	594	396	2470	2069	94	6.11	19.11%
1	CICO 8 VU	#2190	76885	Check-OUT	1273	768	751	913	629	2450	1821	104	5.71	13.54%
2	CICO 8 VU	#2191	76686	Check-IN	941	499	483	607	405	1410	1004	75	4.79	15.03%
2	CICO 8 VU	#2191	76686	Check-OUT	1248	777	754	950	637	2150	1508	109	6.05	14.03%
3	CICO 8 VU	#2192	76559	Check-IN	925	500	481	611	406	2530	2124	97	6.25	19.40%
3	CICO 8 VU	#2192	76559	Check-OUT	1255	778	757	944	640	2030	1393	104	5.75	13.37%
4	CICO 8 VU	#2193	75675	Check-IN	943	492	476	605	404	1480	1080	72	4.60	14.63%
4	CICO 8 VU	#2193	75675	Check-OUT	1221	773	754	938	626	2070	1446	103	5.78	13.32%
5	CICO 8 VU	#2194	76810	Check-IN	916	486	468	586	395	1510	1112	79	5.12	16.26%
5	CICO 8 VU	#2194	76810	Check-OUT	1267	768	743	940	626	2230	1602	121	6.66	15.76%
6	CICO 8 VU	#2195	77187	Check-IN	914	477	465	574	396	1190	791	57	3.70	11.95%
6	CICO 8 VU	#2195	77187	Check-OUT	1278	756	730	922	621	2210	1590	107	5.87	14.15%
7	CICO 8 VU	#2196	76433	Check-IN	924	483	470	581	388	1400	1010	65	4.19	13.46%
7	CICO 8 VU	#2196	76433	Check-OUT	1251	759	738	934	624	2190	1570	102	5.65	13.44%
8	CICO 8 VU	#2197	76460	Check-IN	926	490	479	586	396	1380	980	62	3.99	12.65%
8	CICO 8 VU	#2197	76460	Check-OUT	1252	773	754	929	642	1980	1342	99	5.48	12.81%
9	CICO 8 VU	#2198	76134	Check-IN	951	490	477	590	399	1350	954	66	4.19	13.47%
9	CICO 8 VU	#2198	76134	Check-OUT	1227	773	750	940	645	2100	1451	109	6.10	14.10%
10	CICO 8 VU	#2199	75919	Check-IN	901	488	475	591	404	1480	1080	65	4.24	13.32%
10	CICO 8 VU	#2199	75919	Check-OUT	1255	768	745	929	635	1910	1278	104	5.75	13.54%

where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE=z-score (95% CL=1.96) * StdDev/SQRT(Samples)

CoV - Coefficient of variance - calculated as CoV = StdDev / Mean

		AVG, ms	Median, ms	95pct, ms	StdDev, ms
Check-in	AVG	489.70	475.30	592.50	73.20
	StdDev	6.53	5.57	11.31	12.69
	CoV	1.33%	1.17%	1.91%	17.34%
Check-OUT	AVG	769.30	747.60	933.90	106.20
	StdDev	6.81	8.11	10.39	5.74
	CoV	0.89%	1.09%	1.11%	5.41%

CI/CO 20 VU

							Response time
ID	Test	Job	Total	Request group	Samples	AVG, ms	Median, ms	95pct, ms	Min, ms	Max, ms	Range, ms	StdDev, ms	MoE (95% CL)	CoV
1	CICO 20 VU	#2200	189419	Check-in	2323	481	471	577	322	1480	1158	60	2.44	12.47%
1	CICO 20 VU	#2200	189419	Check-OUT	3082	770	751	922	630	1840	1209	88	3.11	11.43%
2	CICO 20 VU	#2202	189504	Check-IN	2378	488	477	587	341	1560	1220	61	2.45	12.50%
2	CICO 20 VU	#2202	189504	Check-OUT	3048	779	762	933	629	2140	1511	90	3.20	11.55%
3	CICO 20 VU	#2203	190315	Check-IN	2325	485	475	584	334	1460	1126	61	2.48	12.58%
3	CICO 20 VU	#2203	190315	Check-OUT	3103	774	757	933	622	2000	1376	92	3.24	11.89%
4	CICO 20 VU	#2204	189181	Check-IN	2336	489	480	585	341	1470	1126	61	2.47	12.47%
4	CICO 20 VU	#2204	189181	Check-OUT	3069	781	764	935	628	2140	1509	93	3.29	11.91%
5	CICO 20 VU	#2205	189659	Check-IN	2346	496	486	602	346	1420	1073	64	2.59	12.90%
5	CICO 20 VU	#2205	189659	Check-OUT	3074	791	776	956	630	2060	1427	97	3.43	12.26%
6	CICO 20 VU	#2206	190052	Check-IN	2372	471	460	569	329	1520	1187	59	2.37	12.53%
6	CICO 20 VU	#2206	190052	Check-OUT	3068	763	747	923	618	1910	1293	90	3.18	11.80%
7	CICO 20 VU	#2207	190360	Check-IN	2329	471	458	567	333	1460	1129	62	2.52	13.16%
7	CICO 20 VU	#2207	190360	Check-OUT	3101	762	745	924	619	2000	1383	96	3.38	12.60%
8	CICO 20 VU	#2208	191772	Check-IN	2325	472	460	581	316	1460	1140	65	2.64	13.77%
8	CICO 20 VU	#2208	191772	Check-OUT	3138	766	751	932	590	1980	1386	94	3.29	12.27%
9	CICO 20 VU	#2209	188207	Check-IN	2341	480	468	592	323	1480	1157	63	2.55	13.13%
9	CICO 20 VU	#2209	188207	Check-OUT	3040	774	755	940	619	1830	1214	94	3.34	12.14%
10	CICO 20 VU	#2210	191878	Check-IN	2305	477	465	585	323	1100	773	62	2.53	13.00%
10	CICO 20 VU	#2210	191878	Check-OUT	3151	772	754	937	605	2130	1522	98	3.42	12.69%

where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE = z-score (95% CL=1.96) * StdDev/SQRT(Samples)

CoV - Coefficient of variance - calculated as CoV = StdDev / Mean

		AVG, ms	Median, ms	95pct, ms	StdDev, ms
Check-in	AVG	481.00	470.00	582.90	61.80
	StdDev	8.07	8.97	9.79	1.72
	CoV	1.68%	1.91%	1.68%	2.78%
Check-OUT	AVG	773.20	756.20	933.50	93.20
	StdDev	8.40	8.70	9.48	3.09
	CoV	1.09%	1.15%	1.02%	3.32%

1 #2190 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676975322277&to=1676977229179&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=Check-In%20Controller

2 #2191 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676977805847&to=1676979750899&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

3 #2192 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676980200000&to=1676982180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

4 #2193 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676982300000&to=1676984400000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

5 #2194 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676984700000&to=1676986800000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

6 #2195 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676987700000&to=1676989680000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

7 #2196 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676990100000&to=1676992200000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

8 #2197 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676992800000&to=1676994900000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

9 #2198 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676995500000&to=1676997600000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

10 #2199 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676997900000&to=1677000000000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

11 #2200 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677000120000&to=1677002220000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

12 #2202 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677005160000&to=1677007260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

13 #2203 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677007500000&to=1677009480000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

14 #2204 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677009600000&to=1677011820000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

15 #2205 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677012000000&to=1677014100000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

16 #2206 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677049080000&to=1677051180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

17 #2207 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677051600000&to=1677053580000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

18 #2208 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677054300000&to=1677056280000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

19 #2209 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677056820000&to=1677058980000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

20 #2210 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677059220000&to=1677061260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Methodology/Approach

Scripts used:
1. DB Refresh - checkin-checkout-db-restore.sql
2. DB Update - circ-data-load.sh
Execute CI/CO with 8 and 20 concurrent vUsers for 30 minutes 10 times (for each, 20 times in total). (On carrier-io used the artefact "circulation_checkInCheckOut_nolana.zip ")
Gather all needed metrics and their aggregations via the corresponding Grafana dashboard 'JMeter Performance - test Copy'
Calculate the Standard Deviation of the following metrics aggregations: AVG, Median, and 95pct

Browser not supported