MSEARCH-300 - Getting issue details... STATUS

Environment

Summary

Test 1 user call number browsing with random words.

Test 5 users call number browsing with random words.

Test 10 users call number browsing with random words.

Test 20 users call number browsing with random words.

Test 50 users call number browsing with random words.

Test 50 users call number browsing with random words with Data Import in background.
Test 50 users call number browsing with real keywords from database with Data Import in background.
Comparison chart - call number browsing real keys + Data Import in background
Test 50 users call number browsing + DI + CICO

Test with long ramp-up period, 1 minute per user from 0 to 25 users

Environment

mod-search 1.6.4
OpenSearch service
- nodes type r6g.large.search (X4) * (4 data nodes 3 dedicated master nodes);
- version OpenSearch 1.2 (latest);
kafka 2.8.0

Summary

ALL tests (1-50 users) ended without errors. Average response time . most resource intensive is mod-search. During test 50 users call number browsing CPU usage reached more than 300%.
During tests 50 users with background activities (CICO+ Data Import) response time increased up to 10%-15%.

number of users	average rt	% CPU mod-search
1	0.148	12
5	0.168	42
10	0.191	75
20	0.321	122
50	1.821	158

1 user

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	1747	1747	0	0	2.220	0.053	0.142	0.166	0.255	0.308	0.687	0.148	0.251

5 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	8219	8219	0	0	10.693	0.048	0.150	0.188	0.295	0.419	0.920	0.168	0.292

10 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	12741	12741	0	0	20.313	0.051	0.161	0.233	0.387	0.584	1.437	0.191	0.384

20 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	12193	12193	0	0	32.169	0.052	0.245	0.441	0.739	1.025	2.441	0.321	0.736

50 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	7849	7849	0	0	38.485	0.081	0.963	1.259	1.823	2.182	3.275	0.996	1.821

50 users 30 min (10 min ramp up)

50 users + DI in background

50 users +DI 5K in background

RTT without background activity

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers "BT83.57 .N625 2013"	10276	10276	9.349	0.046	0.161	0.234	0.401	0.779	1.547	0.192	0.401
GET_browse _call_numbers "CC76.4 .C66 2013"	10275	10275	9.352	0.046	0.169	0.243	0.402	0.806	1.736	0.199	0.402
GET_browse _call_numbers "DT107.827.L32 1987"	10277	10277	9.346	0.050	0.169	0.242	0.400	0.810	1.721	0.198	0.399
GET_browse _call_numbers "G3893.M5 1923 .W4"	10275	10275	9.336	0.053	0.173	0.248	0.407	0.821	1.753	0.203	0.407
GET_browse _call_numbers "LC4019 .H83 2012"	10275	10275	9.355	0.056	0.181	0.253	0.418	0.823	1.606	0.210	0.418
GET_browse _call_numbers "N6370 .C585 2013eb"	10276	10276	9.343	0.054	0.170	0.243	0.397	0.763	1.847	0.199	0.397
GET_browse _call_numbers "P302.77 .P67 2013"	10273	10273	9.349	0.062	0.192	0.267	0.433	0.808	1.880	0.222	0.433
GET_browse _call_numbers "PR2274 .P7 1633"	10277	10277	9.334	0.047	0.168	0.245	0.411	0.811	1.815	0.201	0.411
GET_browse _call_numbers "TL220 .B85 2012"	10276	10276	9.339	0.043	0.151	0.224	0.383	0.734	1.765	0.182	0.383
GET_browse _call_numbers "Z711 .C88 2013"	10275	10275	9.350	0.093	0.260	0.339	0.507	0.995	2.074	0.289	0.507

RTT with DI in the background.

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers "BT83.57 .N625 2013"	2072	2072	9.280	0.054	0.211	0.293	0.460	0.755	1.423	0.239	0.460
GET_browse _call_numbers "CC76.4 .C66 2013"	2073	2073	9.282	0.051	0.224	0.306	0.469	0.717	1.419	0.249	0.469
GET_browse _call_numbers "DT107.827.L32 1987"	2073	2073	9.277	0.053	0.216	0.297	0.461	0.684	1.359	0.241	0.460
GET_browse _call_numbers "G3893.M5 1923 .W4"	2073	2073	9.270	0.058	0.217	0.302	0.462	0.725	1.332	0.244	0.462
GET_browse _call_numbers "LC4019 .H83 2012"	2071	2071	9.283	0.067	0.234	0.314	0.484	0.742	1.330	0.261	0.484
GET_browse _call_numbers "N6370 .C585 2013eb"	2073	2073	9.275	0.060	0.218	0.298	0.464	0.690	1.454	0.246	0.464
GET_browse _call_numbers "P302.77 .P67 2013"	2071	2071	9.278	0.068	0.240	0.325	0.500	0.752	1.484	0.268	0.500
GET_browse _call_numbers "PR2274 .P7 1633"	2074	2074	9.266	0.049	0.221	0.304	0.464	0.691	1.485	0.245	0.464
GET_browse _call_numbers "TL220 .B85 2012"	2072	2072	9.273	0.044	0.198	0.274	0.438	0.671	1.449	0.225	0.437
GET_browse _call_numbers "Z711 .C88 2013"	2075	2075	9.282	0.103	0.315	0.406	0.591	0.871	1.604	0.340	0.591


average real call numbers from database	0.2095
average random words	0.326

Comparison chart - call number browsing real keys + data import 5K in background

Call number browsing with background CICO activity.

A large usage of processor resources is observed in the mod-search module during call number browsing. I guess need to increase the processor resources in task definition of this module.

Grafana chart from test - random keywords.

Grafana chart from test - real keywords from database.

SELECT jsonb -> effectiveCallNumberComponents →> 'callNumber') FROM fs09000000_mod_inventory_storage.item WHERE jsonb -> effectiveCallNumberComponents →'callNumber' is not null limit 10

50 users call number browsing random words + 10 user CICO in background

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers BT83.57 .N625 2013	5299	5299	8.566	0.048	0.172	0.268	0.475	0.802	2.981	0.214	0.475
GET_browse _call_numbers CC76.4 .C66 2013	5297	5297	8.569	0.050	0.178	0.282	0.486	0.884	2.249	0.224	0.485
GET_browse _call_numbers DT107.827.L32 1987	5297	5297	8.561	0.052	0.181	0.281	0.502	0.922	2.485	0.226	0.502
GET_browse _call_numbers G3893.M5 1923 .W4	5297	5297	8.549	0.054	0.182	0.289	0.511	0.882	2.742	0.230	0.511
GET_browse _call_numbers LC4019 .H83 2012	5298	5298	8.576	0.058	0.193	0.290	0.513	0.879	2.337	0.237	0.513
GET_browse _call_numbers N6370 .C585 2013eb	5296	5296	8.556	0.054	0.178	0.272	0.491	0.869	2.302	0.223	0.491
GET_browse _call_numbers P302.77 .P67 2013	5297	5297	8.562	0.066	0.204	0.310	0.527	0.862	2.916	0.250	0.527
GET_browse _call_numbers PR2274 .P7 1633	5299	5299	8.547	0.048	0.182	0.280	0.504	0.903	2.142	0.228	0.504
GET_browse _call_numbers TL220 .B85 2012	5297	5297	8.552	0.043	0.162	0.259	0.466	0.847	2.224	0.207	0.466
GET_browse _call_numbers Z711 .C88 2013	5300	5300	8.572	0.091	0.262	0.387	0.647	1.157	2.992	0.318	0.647

call number browsing W/O background process

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers BT83.57 .N625 2013	2773	2773	7.757	0.047	0.144	0.223	0.378	0.778	2.292	0.181	0.378
GET_browse _call_numbers CC76.4 .C66 2013	2772	2772	7.762	0.050	0.159	0.240	0.409	0.804	1.594	0.195	0.409
GET_browse _call_numbers DT107.827.L32 1987	2774	2774	7.747	0.050	0.157	0.240	0.402	0.792	2.029	0.193	0.401
GET_browse _call_numbers G3893.M5 1923 .W4	2770	2770	7.726	0.053	0.168	0.253	0.415	0.712	1.655	0.201	0.415
GET_browse _call_numbers LC4019 .H83 2012	2773	2773	7.772	0.059	0.168	0.255	0.415	0.889	1.485	0.206	0.415
GET_browse _call_numbers N6370 .C585 2013eb	2774	2774	7.742	0.054	0.160	0.235	0.402	0.758	1.403	0.195	0.402
GET_browse _call_numbers P302.77 .P67 2013	2773	2773	7.751	0.067	0.186	0.270	0.431	0.797	1.744	0.221	0.431
GET_browse _call_numbers PR2274 .P7 1633	2771	2771	7.720	0.050	0.160	0.240	0.408	0.809	1.845	0.198	0.408
GET_browse _call_numbers TL220 .B85 2012	2776	2776	7.736	0.041	0.140	0.213	0.381	0.816	1.294	0.175	0.381
GET_browse _call_numbers Z711 .C88 2013	2774	2774	7.768	0.090	0.250	0.348	0.526	0.934	1.775	0.284	0.526

Average response times chart (blue- callnumber browsing random words, orange - call number browsing random words + CICO 10 users)_

it can be seen that the response time slightly (15%) increases when working with CICO in background.

long ramp-up period, 1 minute per user from 0 to 25 users

Call number browsing + DI + CICO

Call number browsing with old cluster

1 user

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	2643	2643	0	0	2.233	0.044	0.125	0.164	0.285	0.494	0.948	0.145	0.278

10 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	25569	25569	0	0	21.363	0.043	0.138	0.194	0.350	0.611	1.937	0.167	0.346

20 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	41155	41155	0	0	35.026	0.041	0.196	0.355	0.671	1.078	2.625	0.271	0.669

50 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	103398	103398	0	0	89.437	0.041	0.211	0.313	0.534	0.896	2.146	0.251	0.530

100 users

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
GET_browse _call_numbers	106865	106865	0	0	93.836	0.044	0.727	0.978	1.394	2.073	4.713	0.758	1.389

users	new cluster	old cluster
1	0.148	0.145
10	0.191	0.167
20	0.321	0.271
50	1.821	0.251

Browser not supported