Overview

This document contains the results of testing MOBIUS Baseline Tests: CI/CO. Ticket: PERF-730

PERF-730 - Getting issue details... STATUS

Summary

Three tests were run successfully and without errors on the Mobius-performacne testing cluster (mcpt).
No sign of memory leaks on every module. Memory shows stable trend.
CI/CO for small tenants IDs has a response time of 15% longer than for large tenants IDs.
No modules are showing a growing CPU trend or any spikes.
The CI/CO response time for small tenants IDs with less load is 20% longer than with double load, indicating that CICO works better under a load.

Test Runs

Test №	Date	Test Conditions	Results
1.	Mar 4, 2024, 5:42 PM UTC	1 CI or CO per min for small tenants, 2 CICO per min for large tenants	Completed
2.	Mar 4, 2024, 8:19 PM UTC	2 CI or CO per min for small tenants, 4 CICO per min for large tenants	Completed
3.	Mar 5, 2024, 2:39 PM UTC	4 CICO per min for one large tenant	Completed

Test Results

Test №1

1CI/CO Small + 2CICO Large

Requests	Total per tenant	50th pct	75th pct	95th pct	99th pct	Average
Small tenants - check-in	27	0.992	1.082	1.397	1.831	1.033
Small tenants - checkout	27	1.782	1.964	2.335	2.727	1.831
Large tenants - check-in	92	0.653	0.736	1.024	1.446	0.699
Large tenants - checkout	86	1.217	1.344	1.556	1.894	1.257

Full result table

Requests	Total	50th pct	75th pct	95th pct	99th pct	Average
CICO_TC_Check-Out Controller_cs00000001_0038	25	2.053	2.293	3.029	3.032	2.125
CICO_TC_Check-Out Controller_cs00000001_0032	24	1.952	2.176	2.856	2.945	2.024
CICO_TC_Check-Out Controller_cs00000001_0028	26	1.829	2.081	2.842	3.498	1.953
CICO_TC_Check-Out Controller_cs00000001_0019	24	1.901	2.029	2.554	2.864	1.943
CICO_TC_Check-Out Controller_cs00000001_0016	23	1.807	2.056	2.803	3.414	1.938
CICO_TC_Check-Out Controller_cs00000001_0010	19	1.815	1.967	2.653	2.780	1.933
CICO_TC_Check-Out Controller_cs00000001_0045	26	1.859	2.051	2.359	2.435	1.917
CICO_TC_Check-Out Controller_cs00000001_0046	23	1.817	1.934	2.077	4.394	1.910
CICO_TC_Check-Out Controller_cs00000001_0057	22	1.796	2.141	2.479	2.828	1.896
CICO_TC_Check-Out Controller_cs00000001_0011	29	1.755	2.161	2.672	2.825	1.888
CICO_TC_Check-Out Controller_cs00000001_0049	25	1.823	2.034	2.585	2.615	1.888
CICO_TC_Check-Out Controller_cs00000001_0056	21	1.855	1.948	2.109	2.972	1.885
CICO_TC_Check-Out Controller_cs00000001_0021	22	1.857	1.980	2.475	2.777	1.881
CICO_TC_Check-Out Controller_cs00000001_0029	27	1.891	1.969	2.406	2.595	1.871
CICO_TC_Check-Out Controller_cs00000001_0052	25	1.842	2.037	2.153	2.513	1.868
CICO_TC_Check-Out Controller_cs00000001_0055	20	1.931	2.032	2.220	2.355	1.868
CICO_TC_Check-Out Controller_cs00000001_0039	22	1.817	1.993	2.479	2.608	1.863
CICO_TC_Check-Out Controller_cs00000001_0005	25	1.851	2.034	2.300	2.357	1.858
CICO_TC_Check-Out Controller_cs00000001_0044	25	1.879	1.974	2.298	2.591	1.857
CICO_TC_Check-Out Controller_cs00000001_0020	24	1.813	2.023	2.205	3.641	1.855
CICO_TC_Check-Out Controller_cs00000001_0026	25	1.733	1.965	2.574	3.367	1.855
CICO_TC_Check-Out Controller_cs00000001_0033	24	1.864	1.994	2.120	2.329	1.848
CICO_TC_Check-Out Controller_cs00000001_0025	24	1.798	1.908	2.716	2.795	1.840
CICO_TC_Check-Out Controller_cs00000001_0004	29	1.806	2.030	2.352	2.668	1.833
CICO_TC_Check-Out Controller_cs00000001_0023	29	1.860	2.026	2.313	2.386	1.831
CICO_TC_Check-Out Controller_cs00000001_0031	25	1.814	2.039	2.161	2.788	1.831
CICO_TC_Check-Out Controller_cs00000001_0060	34	1.704	2.019	2.549	2.799	1.828
CICO_TC_Check-Out Controller_cs00000001_0030	32	1.714	2.037	2.581	2.758	1.825
CICO_TC_Check-Out Controller_cs00000001_0018	22	1.815	1.932	2.222	2.353	1.819
CICO_TC_Check-Out Controller_cs00000001_0040	30	1.768	1.982	2.353	2.539	1.817
CICO_TC_Check-Out Controller_cs00000001_0002	23	1.846	1.930	2.054	2.681	1.816
CICO_TC_Check-Out Controller_cs00000001_0034	28	1.730	1.853	2.539	2.597	1.815
CICO_TC_Check-Out Controller_cs00000001_0012	29	1.705	1.871	2.703	3.061	1.814
CICO_TC_Check-Out Controller_cs00000001_0007	33	1.760	1.995	2.148	2.626	1.812
CICO_TC_Check-Out Controller_cs00000001_0022	32	1.704	1.873	2.227	3.489	1.810
CICO_TC_Check-Out Controller_cs00000001_0041	27	1.754	1.953	2.204	2.757	1.809
CICO_TC_Check-Out Controller_cs00000001_0015	23	1.784	1.940	2.156	2.259	1.801
CICO_TC_Check-Out Controller_cs00000001_0036	20	1.832	1.942	2.066	2.162	1.800
CICO_TC_Check-Out Controller_cs00000001_0048	31	1.718	1.850	2.325	2.784	1.799
CICO_TC_Check-Out Controller_cs00000001_0017	25	1.780	2.003	2.206	2.320	1.798
CICO_TC_Check-Out Controller_cs00000001_0061	32	1.724	1.965	2.170	2.800	1.797
CICO_TC_Check-Out Controller_cs00000001_0047	27	1.734	1.958	2.316	2.522	1.795
CICO_TC_Check-Out Controller_cs00000001_0008	24	1.813	1.948	2.170	2.257	1.785
CICO_TC_Check-Out Controller_cs00000001_0027	31	1.703	1.877	2.311	3.142	1.785
CICO_TC_Check-Out Controller_cs00000001_0001	26	1.683	1.919	2.145	2.794	1.777
CICO_TC_Check-Out Controller_cs00000001_0050	29	1.794	1.867	2.301	2.940	1.776
CICO_TC_Check-Out Controller_cs00000001_0003	27	1.636	1.901	2.246	3.410	1.773
CICO_TC_Check-Out Controller_cs00000001_0062	27	1.744	1.897	2.161	2.163	1.765
CICO_TC_Check-Out Controller_cs00000001_0009	32	1.734	1.832	2.159	2.930	1.764
CICO_TC_Check-Out Controller_cs00000001_0014	30	1.718	1.867	2.178	2.654	1.762
CICO_TC_Check-Out Controller_cs00000001_0035	30	1.672	1.866	2.417	2.623	1.756
CICO_TC_Check-Out Controller_cs00000001_0053	29	1.771	1.872	2.146	2.331	1.752
CICO_TC_Check-Out Controller_cs00000001_0059	27	1.637	1.907	2.115	2.541	1.741
CICO_TC_Check-Out Controller_cs00000001_0058	32	1.655	1.796	2.147	2.764	1.738
CICO_TC_Check-Out Controller_cs00000001_0054	28	1.708	1.931	2.046	2.089	1.734
CICO_TC_Check-Out Controller_cs00000001_0043	31	1.663	1.896	2.013	2.124	1.733
CICO_TC_Check-Out Controller_cs00000001_0013	28	1.689	1.834	2.301	2.509	1.703
CICO_TC_Check-Out Controller_cs00000001_0037	36	1.643	1.713	1.955	2.043	1.651
CICO_TC_Check-Out Controller_cs00000001_0006	82	1.250	1.355	1.584	1.860	1.280
CICO_TC_Check-Out Controller_cs00000001_0024	82	1.192	1.321	1.521	1.616	1.222
CICO_TC_Check-Out Controller_cs00000001_0042	90	1.229	1.341	1.550	1.980	1.275
CICO_TC_Check-Out Controller_cs00000001_0051	90	1.198	1.359	1.569	2.121	1.249
CICO_TC_Check-In Controller_cs00000001_0038	29	1.287	1.378	1.939	2.149	1.278
CICO_TC_Check-In Controller_cs00000001_0018	33	0.955	1.133	1.783	3.462	1.129
CICO_TC_Check-In Controller_cs00000001_0015	32	1.024	1.080	1.666	2.446	1.120
CICO_TC_Check-In Controller_cs00000001_0019	30	1.014	1.125	1.736	3.129	1.112
CICO_TC_Check-In Controller_cs00000001_0001	29	1.030	1.134	1.790	1.844	1.105
CICO_TC_Check-In Controller_cs00000001_0050	25	0.968	1.161	1.686	1.862	1.105
CICO_TC_Check-In Controller_cs00000001_0053	24	1.005	1.139	1.690	2.365	1.096
CICO_TC_Check-In Controller_cs00000001_0021	32	1.013	1.119	1.410	2.925	1.094
CICO_TC_Check-In Controller_cs00000001_0035	24	1.002	1.077	1.171	3.426	1.084
CICO_TC_Check-In Controller_cs00000001_0039	32	1.038	1.171	1.301	1.677	1.074
CICO_TC_Check-In Controller_cs00000001_0058	21	1.026	1.135	1.748	1.753	1.073
CICO_TC_Check-In Controller_cs00000001_0014	24	0.998	1.105	1.699	1.746	1.071
CICO_TC_Check-In Controller_cs00000001_0054	26	0.961	1.101	1.698	2.320	1.070
CICO_TC_Check-In Controller_cs00000001_0036	34	0.992	1.078	1.363	2.987	1.069
CICO_TC_Check-In Controller_cs00000001_0047	27	1.036	1.094	1.234	1.716	1.051
CICO_TC_Check-In Controller_cs00000001_0055	34	1.002	1.102	1.310	1.782	1.049
CICO_TC_Check-In Controller_cs00000001_0040	24	1.010	1.047	1.533	1.606	1.048
CICO_TC_Check-In Controller_cs00000001_0037	18	1.026	1.124	1.329	1.385	1.044
CICO_TC_Check-In Controller_cs00000001_0029	27	1.008	1.104	1.293	1.569	1.043
CICO_TC_Check-In Controller_cs00000001_0030	22	0.984	1.048	1.288	1.912	1.040
CICO_TC_Check-In Controller_cs00000001_0034	26	1.000	1.095	1.731	1.774	1.038
CICO_TC_Check-In Controller_cs00000001_0009	23	1.019	1.083	1.362	1.387	1.037
CICO_TC_Check-In Controller_cs00000001_0005	29	0.968	1.063	1.716	2.030	1.036
CICO_TC_Check-In Controller_cs00000001_0043	23	0.960	1.071	1.697	1.710	1.029
CICO_TC_Check-In Controller_cs00000001_0011	26	0.976	1.086	1.735	1.806	1.026
CICO_TC_Check-In Controller_cs00000001_0033	30	1.008	1.137	1.405	1.419	1.025
CICO_TC_Check-In Controller_cs00000001_0041	27	1.001	1.099	1.260	1.412	1.025
CICO_TC_Check-In Controller_cs00000001_0056	33	1.005	1.049	1.315	1.754	1.025
CICO_TC_Check-In Controller_cs00000001_0010	35	0.991	1.037	1.265	1.772	1.022
CICO_TC_Check-In Controller_cs00000001_0046	31	0.973	1.107	1.226	1.695	1.022
CICO_TC_Check-In Controller_cs00000001_0044	29	1.028	1.074	1.299	1.638	1.021
CICO_TC_Check-In Controller_cs00000001_0045	28	0.951	1.017	1.619	1.683	1.021
CICO_TC_Check-In Controller_cs00000001_0027	23	0.984	1.058	1.211	1.797	1.020
CICO_TC_Check-In Controller_cs00000001_0013	27	0.921	1.080	1.759	1.804	1.019
CICO_TC_Check-In Controller_cs00000001_0008	31	1.023	1.037	1.227	2.076	1.018
CICO_TC_Check-In Controller_cs00000001_0023	25	1.003	1.048	1.478	1.699	1.018
CICO_TC_Check-In Controller_cs00000001_0062	26	0.999	1.049	1.248	1.852	1.018
CICO_TC_Check-In Controller_cs00000001_0003	27	0.999	1.086	1.171	1.247	1.010
CICO_TC_Check-In Controller_cs00000001_0002	32	0.974	1.052	1.223	2.133	1.009
CICO_TC_Check-In Controller_cs00000001_0028	28	0.958	1.106	1.298	1.841	1.006
CICO_TC_Check-In Controller_cs00000001_0049	28	0.984	1.066	1.176	1.524	1.006
CICO_TC_Check-In Controller_cs00000001_0004	25	0.968	1.049	1.299	1.810	1.001
CICO_TC_Check-In Controller_cs00000001_0048	23	0.928	1.048	1.559	1.746	1.000
CICO_TC_Check-In Controller_cs00000001_0017	30	1.010	1.060	1.164	1.659	0.996
CICO_TC_Check-In Controller_cs00000001_0016	32	0.990	1.038	1.153	1.360	0.994
CICO_TC_Check-In Controller_cs00000001_0025	30	0.999	1.035	1.228	1.377	0.994
CICO_TC_Check-In Controller_cs00000001_0026	29	0.979	1.019	1.271	1.741	0.991
CICO_TC_Check-In Controller_cs00000001_0057	31	0.970	1.051	1.129	1.810	0.990
CICO_TC_Check-In Controller_cs00000001_0020	30	0.999	1.063	1.220	1.314	0.989
CICO_TC_Check-In Controller_cs00000001_0032	30	0.956	1.052	1.334	1.786	0.988
CICO_TC_Check-In Controller_cs00000001_0022	22	0.993	1.092	1.188	1.209	0.985
CICO_TC_Check-In Controller_cs00000001_0059	27	0.979	1.058	1.182	1.220	0.983
CICO_TC_Check-In Controller_cs00000001_0060	20	0.968	1.055	1.117	1.415	0.983
CICO_TC_Check-In Controller_cs00000001_0012	26	0.980	1.048	1.167	1.319	0.981
CICO_TC_Check-In Controller_cs00000001_0007	22	0.936	1.024	1.165	1.756	0.978
CICO_TC_Check-In Controller_cs00000001_0061	21	0.904	1.003	1.467	1.783	0.974
CICO_TC_Check-In Controller_cs00000001_0031	29	0.965	1.055	1.131	1.177	0.953
CICO_TC_Check-In Controller_cs00000001_0052	29	0.914	1.022	1.177	1.622	0.943
CICO_TC_Check-In Controller_cs00000001_0024	96	0.677	0.755	1.232	1.424	0.729
CICO_TC_Check-In Controller_cs00000001_0042	88	0.659	0.733	0.960	1.217	0.700
CICO_TC_Check-In Controller_cs00000001_0051	88	0.633	0.715	1.053	1.627	0.684
CICO_TC_Check-In Controller_cs00000001_0006	97	0.641	0.742	0.851	1.517	0.682
CICO_POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	1898	0.455	0.515	0.591	0.746	0.463
CICO_POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	1969	0.418	0.466	0.613	0.941	0.438

Test №2

2CI/CO Small + 4CICO Large

Requests	Total per tenant	50th pct	75th pct	95th pct	99th pct	Average
Small tenants - check-in	52	0.755	0.863	1.089	1.474	0.802
Small tenants - checkout	51	1.437	1.598	1.940	2.263	1.473
Large tenants - check-in	141	0.585	0.653	0.842	1.225	0.626
Large tenants - checkout	132	1.108	1.233	1.455	2.166	1.153

Full result table

Requests	Total	50th pct	75th pct	95th pct	99th pct	Average
CICO_TC_Check-Out Controller_cs00000001_0038	58	1.542	1.807	2.383	2.612	1.603
CICO_TC_Check-Out Controller_cs00000001_0037	42	1.514	1.664	2.288	2.418	1.549
CICO_TC_Check-Out Controller_cs00000001_0005	50	1.483	1.674	2.257	2.773	1.541
CICO_TC_Check-Out Controller_cs00000001_0023	46	1.478	1.689	2.155	2.981	1.537
CICO_TC_Check-Out Controller_cs00000001_0032	48	1.434	1.684	2.205	4.499	1.537
CICO_TC_Check-Out Controller_cs00000001_0020	52	1.431	1.557	2.225	2.876	1.531
CICO_TC_Check-Out Controller_cs00000001_0009	43	1.528	1.653	1.910	2.362	1.518
CICO_TC_Check-Out Controller_cs00000001_0035	41	1.464	1.642	1.985	2.567	1.518
CICO_TC_Check-Out Controller_cs00000001_0018	49	1.402	1.682	2.387	2.738	1.516
CICO_TC_Check-Out Controller_cs00000001_0031	54	1.431	1.609	2.028	2.382	1.504
CICO_TC_Check-Out Controller_cs00000001_0022	46	1.436	1.597	2.198	2.461	1.503
CICO_TC_Check-Out Controller_cs00000001_0017	46	1.466	1.670	2.046	2.124	1.502
CICO_TC_Check-Out Controller_cs00000001_0028	52	1.421	1.592	2.217	2.358	1.500
CICO_TC_Check-Out Controller_cs00000001_0026	51	1.454	1.616	2.018	2.271	1.499
CICO_TC_Check-Out Controller_cs00000001_0034	51	1.461	1.624	1.893	2.148	1.499
CICO_TC_Check-Out Controller_cs00000001_0060	48	1.464	1.614	1.906	2.715	1.492
CICO_TC_Check-Out Controller_cs00000001_0048	54	1.425	1.613	2.170	2.336	1.490
CICO_TC_Check-Out Controller_cs00000001_0029	50	1.453	1.627	1.914	2.494	1.486
CICO_TC_Check-Out Controller_cs00000001_0058	51	1.423	1.601	1.936	2.282	1.484
CICO_TC_Check-Out Controller_cs00000001_0001	53	1.503	1.588	1.679	2.045	1.483
CICO_TC_Check-Out Controller_cs00000001_0019	53	1.422	1.612	2.211	2.319	1.482
CICO_TC_Check-Out Controller_cs00000001_0025	55	1.406	1.622	2.255	2.451	1.481
CICO_TC_Check-Out Controller_cs00000001_0033	44	1.470	1.610	2.013	2.069	1.480
CICO_TC_Check-Out Controller_cs00000001_0047	44	1.427	1.657	1.829	2.067	1.478
CICO_TC_Check-Out Controller_cs00000001_0040	52	1.446	1.564	1.864	2.301	1.477
CICO_TC_Check-Out Controller_cs00000001_0010	50	1.462	1.555	2.035	2.388	1.476
CICO_TC_Check-Out Controller_cs00000001_0056	52	1.442	1.639	1.816	2.115	1.473
CICO_TC_Check-Out Controller_cs00000001_0002	59	1.449	1.607	1.911	2.172	1.472
CICO_TC_Check-Out Controller_cs00000001_0059	52	1.419	1.557	1.704	1.998	1.470
CICO_TC_Check-Out Controller_cs00000001_0053	51	1.455	1.582	1.733	1.959	1.469
CICO_TC_Check-Out Controller_cs00000001_0062	48	1.469	1.632	1.813	1.988	1.468
CICO_TC_Check-Out Controller_cs00000001_0045	49	1.456	1.588	1.856	2.253	1.467
CICO_TC_Check-Out Controller_cs00000001_0039	51	1.471	1.607	1.815	1.923	1.466
CICO_TC_Check-Out Controller_cs00000001_0054	53	1.449	1.583	1.727	2.100	1.466
CICO_TC_Check-Out Controller_cs00000001_0044	53	1.425	1.560	1.772	2.227	1.462
CICO_TC_Check-Out Controller_cs00000001_0016	48	1.446	1.583	1.794	2.349	1.461
CICO_TC_Check-Out Controller_cs00000001_0046	64	1.391	1.544	2.231	2.377	1.459
CICO_TC_Check-Out Controller_cs00000001_0057	52	1.391	1.581	1.768	1.891	1.455
CICO_TC_Check-Out Controller_cs00000001_0011	46	1.464	1.558	1.838	1.895	1.452
CICO_TC_Check-Out Controller_cs00000001_0036	48	1.428	1.631	1.893	2.052	1.451
CICO_TC_Check-Out Controller_cs00000001_0052	56	1.423	1.550	1.922	2.321	1.451
CICO_TC_Check-Out Controller_cs00000001_0015	54	1.414	1.527	1.803	1.980	1.447
CICO_TC_Check-Out Controller_cs00000001_0008	48	1.449	1.594	1.875	2.138	1.445
CICO_TC_Check-Out Controller_cs00000001_0012	53	1.422	1.570	1.911	2.106	1.445
CICO_TC_Check-Out Controller_cs00000001_0061	57	1.388	1.585	1.820	2.454	1.445
CICO_TC_Check-Out Controller_cs00000001_0003	56	1.411	1.533	1.984	2.243	1.443
CICO_TC_Check-Out Controller_cs00000001_0027	56	1.414	1.576	1.810	1.956	1.443
CICO_TC_Check-Out Controller_cs00000001_0004	44	1.470	1.588	1.778	1.837	1.441
CICO_TC_Check-Out Controller_cs00000001_0050	47	1.441	1.628	1.823	1.877	1.441
CICO_TC_Check-Out Controller_cs00000001_0021	53	1.410	1.544	1.773	2.237	1.440
CICO_TC_Check-Out Controller_cs00000001_0041	51	1.401	1.533	1.854	1.885	1.440
CICO_TC_Check-Out Controller_cs00000001_0014	50	1.450	1.582	1.809	1.959	1.436
CICO_TC_Check-Out Controller_cs00000001_0049	56	1.391	1.572	1.839	2.169	1.428
CICO_TC_Check-Out Controller_cs00000001_0055	53	1.389	1.529	1.719	1.753	1.420
CICO_TC_Check-Out Controller_cs00000001_0043	51	1.398	1.521	1.665	1.730	1.418
CICO_TC_Check-Out Controller_cs00000001_0007	59	1.380	1.560	1.807	2.225	1.414
CICO_TC_Check-Out Controller_cs00000001_0030	61	1.401	1.544	1.671	1.775	1.408
CICO_TC_Check-Out Controller_cs00000001_0013	50	1.306	1.449	1.960	2.281	1.345
CICO_TC_Check-Out Controller_cs00000001_0042	125	1.102	1.229	1.467	2.682	1.164
CICO_TC_Check-Out Controller_cs00000001_0024	125	1.119	1.252	1.452	2.039	1.157
CICO_TC_Check-Out Controller_cs00000001_0051	129	1.125	1.232	1.479	1.949	1.154
CICO_TC_Check-Out Controller_cs00000001_0006	150	1.086	1.220	1.423	1.993	1.135
CICO_TC_Check-In Controller_cs00000001_0038	46	0.847	1.019	1.191	1.306	0.875
CICO_TC_Check-In Controller_cs00000001_0021	50	0.776	0.910	1.452	2.977	0.855
CICO_TC_Check-In Controller_cs00000001_0008	56	0.760	0.849	0.984	2.024	0.854
CICO_TC_Check-In Controller_cs00000001_0028	51	0.784	0.926	1.225	1.708	0.853
CICO_TC_Check-In Controller_cs00000001_0032	55	0.723	0.900	1.621	1.680	0.850
CICO_TC_Check-In Controller_cs00000001_0049	47	0.768	0.920	1.304	1.537	0.848
CICO_TC_Check-In Controller_cs00000001_0003	48	0.773	0.881	1.661	2.219	0.844
CICO_TC_Check-In Controller_cs00000001_0009	61	0.782	0.895	1.090	1.501	0.835
CICO_TC_Check-In Controller_cs00000001_0055	50	0.754	0.882	1.249	1.715	0.828
CICO_TC_Check-In Controller_cs00000001_0004	60	0.754	0.866	1.341	1.900	0.826
CICO_TC_Check-In Controller_cs00000001_0026	53	0.759	0.877	1.058	1.766	0.825
CICO_TC_Check-In Controller_cs00000001_0016	55	0.729	0.807	1.447	1.701	0.821
CICO_TC_Check-In Controller_cs00000001_0030	42	0.766	0.939	1.096	1.779	0.820
CICO_TC_Check-In Controller_cs00000001_0014	54	0.761	0.885	1.096	1.650	0.819
CICO_TC_Check-In Controller_cs00000001_0018	54	0.775	0.894	1.042	1.089	0.816
CICO_TC_Check-In Controller_cs00000001_0034	52	0.782	0.854	1.046	1.544	0.814
CICO_TC_Check-In Controller_cs00000001_0019	51	0.736	0.893	1.106	1.567	0.813
CICO_TC_Check-In Controller_cs00000001_0017	58	0.727	0.911	1.174	1.388	0.810
CICO_TC_Check-In Controller_cs00000001_0025	49	0.785	0.862	1.007	1.803	0.810
CICO_TC_Check-In Controller_cs00000001_0040	51	0.777	0.888	1.056	1.262	0.810
CICO_TC_Check-In Controller_cs00000001_0052	47	0.770	0.861	1.110	1.543	0.808
CICO_TC_Check-In Controller_cs00000001_0047	59	0.750	0.858	1.388	1.541	0.807
CICO_TC_Check-In Controller_cs00000001_0011	58	0.753	0.841	0.950	1.817	0.806
CICO_TC_Check-In Controller_cs00000001_0059	51	0.769	0.858	1.065	1.230	0.804
CICO_TC_Check-In Controller_cs00000001_0012	51	0.758	0.826	1.011	1.449	0.803
CICO_TC_Check-In Controller_cs00000001_0045	54	0.761	0.872	1.048	1.623	0.803
CICO_TC_Check-In Controller_cs00000001_0046	39	0.769	0.865	1.114	1.493	0.802
CICO_TC_Check-In Controller_cs00000001_0057	51	0.785	0.873	1.019	1.048	0.802
CICO_TC_Check-In Controller_cs00000001_0035	63	0.748	0.844	1.005	1.782	0.798
CICO_TC_Check-In Controller_cs00000001_0029	54	0.734	0.847	1.166	1.491	0.797
CICO_TC_Check-In Controller_cs00000001_0007	45	0.741	0.845	1.054	1.679	0.796
CICO_TC_Check-In Controller_cs00000001_0015	49	0.744	0.858	0.960	2.005	0.795
CICO_TC_Check-In Controller_cs00000001_0041	53	0.753	0.845	0.979	1.261	0.794
CICO_TC_Check-In Controller_cs00000001_0023	58	0.737	0.846	1.004	1.545	0.793
CICO_TC_Check-In Controller_cs00000001_0039	53	0.751	0.821	1.035	1.588	0.793
CICO_TC_Check-In Controller_cs00000001_0043	53	0.764	0.881	1.005	1.219	0.790
CICO_TC_Check-In Controller_cs00000001_0036	55	0.698	0.846	1.003	1.589	0.789
CICO_TC_Check-In Controller_cs00000001_0053	52	0.765	0.876	1.016	1.051	0.787
CICO_TC_Check-In Controller_cs00000001_0054	50	0.739	0.841	1.186	1.536	0.787
CICO_TC_Check-In Controller_cs00000001_0048	49	0.732	0.866	1.033	1.571	0.786
CICO_TC_Check-In Controller_cs00000001_0050	56	0.749	0.839	1.020	1.307	0.784
CICO_TC_Check-In Controller_cs00000001_0010	54	0.743	0.841	1.002	1.327	0.783
CICO_TC_Check-In Controller_cs00000001_0031	50	0.750	0.862	1.048	1.586	0.782
CICO_TC_Check-In Controller_cs00000001_0033	60	0.740	0.826	1.025	1.142	0.782
CICO_TC_Check-In Controller_cs00000001_0037	61	0.768	0.827	0.986	1.074	0.782
CICO_TC_Check-In Controller_cs00000001_0002	45	0.764	0.808	1.028	1.526	0.781
CICO_TC_Check-In Controller_cs00000001_0044	50	0.751	0.832	0.949	1.618	0.776
CICO_TC_Check-In Controller_cs00000001_0061	46	0.749	0.859	1.020	1.053	0.775
CICO_TC_Check-In Controller_cs00000001_0013	54	0.749	0.811	0.926	1.166	0.772
CICO_TC_Check-In Controller_cs00000001_0056	51	0.765	0.829	0.984	1.060	0.772
CICO_TC_Check-In Controller_cs00000001_0022	57	0.728	0.856	1.003	1.107	0.771
CICO_TC_Check-In Controller_cs00000001_0005	54	0.743	0.874	0.986	1.078	0.770
CICO_TC_Check-In Controller_cs00000001_0020	51	0.741	0.847	0.963	1.004	0.768
CICO_TC_Check-In Controller_cs00000001_0001	51	0.732	0.801	1.009	1.299	0.767
CICO_TC_Check-In Controller_cs00000001_0060	55	0.753	0.850	0.939	0.981	0.767
CICO_TC_Check-In Controller_cs00000001_0058	52	0.734	0.851	0.989	1.065	0.766
CICO_TC_Check-In Controller_cs00000001_0062	55	0.746	0.859	0.953	0.970	0.763
CICO_TC_Check-In Controller_cs00000001_0027	47	0.745	0.852	0.923	0.976	0.761
CICO_TC_Check-In Controller_cs00000001_0051	144	0.582	0.645	0.834	1.072	0.631
CICO_TC_Check-In Controller_cs00000001_0024	148	0.583	0.661	0.837	1.300	0.627
CICO_TC_Check-In Controller_cs00000001_0006	123	0.589	0.665	0.844	1.023	0.623
CICO_TC_Check-In Controller_cs00000001_0042	148	0.586	0.639	0.851	1.504	0.622
CICO_POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	3499	0.398	0.457	0.597	0.765	0.419
CICO_POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	3610	0.355	0.408	0.548	1.030	0.382

Test №3

4CICO Large

Requests	Total per tenant	50th pct	75th pct	95th pct	99th pct	Average
CICO_TC_Check-Out Controller_cs00000001_0006	130	1.161	1.257	1.479	2.618	1.216
CICO_TC_Check-In Controller_cs00000001_0006	143	0.611	0.673	0.771	0.861	0.619

Full result table

Requests	Total	50th pct	75th pct	95th pct	99th pct	Average
CICO_TC_Check-Out Controller_cs00000001_0006	130	1.161	1.257	1.479	2.618	1.216
CICO_TC_Check-In Controller_cs00000001_0006	143	0.611	0.673	0.771	0.861	0.619
CICO_POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	130	0.349	0.393	0.521	1.067	0.384
CICO_POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	144	0.296	0.336	0.416	0.462	0.309

Comparison

The following diagram compares all test results of MOBIUS Baseline Tests: CI/CO.

Resource utilization for Test #1

Resource utilization table

CPU		Memory
mod-authtoken-b	33.03	okapi-b	73.43
mod-users-b	23.36	mod-inventory-b	64.56
okapi-b	16.59	mod-data-import-b	58.45
mod-configuration-b	9.41	mod-pubsub-b	47.46
mod-feesfines-b	9.22	mod-circulation-b	45.97
mod-pubsub-b	11.72	mod-users-b	37.22
mod-inventory-b	10.10	mod-patron-blocks-b	34.15
mod-circulation-b	2.03	mod-feesfines-b	32.86
mod-quick-marc-b	7.53	mod-configuration-b	31.38
mod-circulation-storage-b	2.17	mod-source-record-storage-b	31.20
mod-inventory-storage-b	3.14	mod-quick-marc-b	27.80
mod-source-record-storage-b	2.65	mod-authtoken-b	27.56
mod-patron-blocks-b	1.56	mod-circulation-storage-b	26.75
mod-source-record-manager-b	1.72	mod-users-bl-b	23.22
mod-remote-storage-b	1.17	mod-source-record-manager-b	23.00
mod-data-import-b	1.32	mod-remote-storage-b	21.15
mod-users-bl-b	0.64	mod-inventory-storage-b	19.21

Service CPU Utilization

Here we can see mod-authtoken, mod-users, okapi modules using the most resources.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 7% in average and 11% during spike.

DB Connections

DB connections was 1800 in average and 2400 during spike.

DB load

Top SQL-queries

Resource utilization for Test #2

Resource utilization table

CPU		Memory
mod-authtoken-b	43.94	okapi-b	74.05
mod-users-b	34.06	mod-inventory-b	64.61
okapi-b	30.00	mod-data-import-b	58.41
mod-configuration-b	22.67	mod-pubsub-b	48.11
mod-feesfines-b	21.80	mod-circulation-b	41.83
mod-pubsub-b	12.69	mod-users-b	37.59
mod-inventory-b	10.98	mod-patron-blocks-b	34.55
mod-circulation-b	9.85	mod-feesfines-b	33.15
mod-quick-marc-b	8.11	mod-configuration-b	31.64
mod-circulation-storage-b	7.64	mod-source-record-storage-b	31.21
mod-inventory-storage-b	5.09	mod-quick-marc-b	27.80
mod-source-record-storage-b	2.70	mod-authtoken-b	27.47
mod-patron-blocks-b	1.86	mod-circulation-storage-b	26.94
mod-source-record-manager-b	1.76	mod-users-bl-b	23.39
mod-remote-storage-b	1.59	mod-source-record-manager-b	22.99
mod-data-import-b	1.46	mod-remote-storage-b	21.22
mod-users-bl-b	0.61	mod-inventory-storage-b	19.33

Service CPU Utilization

Here we can see mod-authtoken, mod-users, okapi modules using the most resources.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 10% in average and 14% during spike.

DB Connections

DB connections was 2100.

DB load

Top SQL-queries

Resource utilization for Test #3

Resource utilization table

CPU		Memory
mod-configuration-b	12.19	mod-inventory-b	93.75
mod-inventory-b	9.55	okapi-b	72.78
mod-pubsub-b	9.45	mod-source-record-manager-b	58.04
mod-users-b	8.50	mod-data-import-b	54.34
mod-circulation-b	8.11	mod-pubsub-b	44.68
okapi-b	7.19	mod-patron-blocks-b	35.99
mod-feesfines-b	6.70	mod-users-b	35.68
mod-quick-marc-b	6.11	mod-source-record-storage-b	34.91
mod-circulation-storage-b	5.18	mod-circulation-b	34.50
mod-authtoken-b	4.43	mod-feesfines-b	32.89
mod-source-record-manager-b	3.71	mod-configuration-b	31.18
mod-source-record-storage-b	2.16	mod-quick-marc-b	27.76
mod-data-import-b	1.57	mod-authtoken-b	26.17
mod-patron-blocks-b	0.98	mod-circulation-storage-b	22.10
mod-remote-storage-b	0.74	mod-inventory-storage-b	20.99
mod-users-bl-b	0.54	mod-users-bl-b	20.34
mod-inventory-storage-b	0.39	mod-remote-storage-b	17.06

Service CPU Utilization

Here we can see that all modules show a stable trend.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 3% in average and 6.5% during spike.

DB Connections

DB connections was 900.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - environment mcpt

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer
Name Memory GIB vCPUs
db.r6g.4xlarge
128 GiB 16 vCPUs
MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2

Modules memory and CPU parameters:

Module mcpt-pvt Wed Mar 06 14:04:07 UTC 2024	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-inventory-storage	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
mod-inventory	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.1.6	2	2880	2592	1024	1814	384	512	FALSE
okapi-b	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-source-record-storage	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-source-record-manager	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512	FALSE
mod-data-import	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-quick-marc	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
mod-feesfines	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.0.0	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	FALSE
mod-pubsub	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512	FALSE
mod-authtoken	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.1	2	1440	1152	512	922	88	128	FALSE
mod-circulation-storage	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	FALSE
mod-circulation	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512	FALSE
mod-configuration	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-remote-storage	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512	FALSE
mod-users-bl	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE

Methodology/Approach

MOBIUS Baseline Tests: CI/CO scenarios were started by Jenkins.

The database contains five times more generated data for large tenant IDs compared to small tenant IDs. The table below displays the average count of generated instances and items for both small and large tenants..

tenant	instances	items
Small	84754	103137
Large	416176	543428

The attached file contains full info about generated data per tenant.

large tenants IDs = 0042, 0024, 0051, 0006

Run data preparation script before each CI/CO test
Create artefacts and upload to carrier-io
Use Jenkins job to change parameters and run tests
Executed one check-in or check-out request per minute concurrently for 58 small tenants, plus 2 check-in and check-out requests per minute concurrently for 4 large tenants. Totally: 5 requests per min
Executed 2 check-in or check-out request per minute concurrently for 58 small tenants, plus 4 check-in and check-out requests per minute concurrently for 4 large tenants. Totally: 10 requests per min
Executed 4 check-in and check-out requests per minute for one large tenant. Totally: 8 requests per min

Folio Development Teams

MOBIUS Baseline Tests: CI/CO multi tenants

Overview

Summary

Test Runs

Test Results

Test №1

Test №2

Test №3

Comparison

Resource utilization for Test #1

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #2

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test #3

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Methodology/Approach

Additional Screenshots of graphs or charts

Name	Memory GIB	vCPUs
db.r6g.4xlarge	128 GiB	16 vCPUs