Table of Contents

Overview

In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release -

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-225

...

7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
80 FOLIO back-end modules deployed in 166 ECS services
3 okapi ECS services
7 m5.xlarge EC2 instances
1 writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

...

	1 user - Baseline (avg) ms	1 user - Under the load (avg) ms	20 user - Under the load (avg) ms	20 user - Under the load (avg) ms
Total requests processed in 60 minutes test run	10715	8051	69244	59420

Test Runs

1 User without any background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	233	0.065	0.250	0.264	0.298	0.520	0.721	0.973	0.302	0.520
fs09000000GET /search/authorities keyword <> randomword	233	0.065	0.246	0.261	0.289	0.548	0.810	1.771	0.312	0.548
fs09000000GET /search/authorities keyword = random sentence	233	0.065	0.567	0.710	0.962	1.681	2.240	2.582	0.874	1.681
fs09000000GET /search/authorities keyword = random sentence	233	0.065	0.017	0.021	0.023	0.034	0.048	0.074	0.023	0.034
fs09000000GET /search/authorities keyword = randomword	233	0.065	0.014	0.024	0.056	0.135	0.204	0.449	0.044	0.135
fs09000000GET /search/authorities keyword == random sentence	233	0.065	0.017	0.020	0.022	0.035	0.055	0.101	0.022	0.033
fs09000000GET /search/authorities keyword == randomword	233	0.065	0.015	0.022	0.034	0.089	0.164	0.241	0.034	0.089
fs09000000GET /search/authorities keyword all randomword	233	0.065	0.302	0.554	0.824	1.198	1.675	2.013	0.644	1.198
fs09000000GET /search/authorities keyword all random sentence	233	0.065	0.017	0.021	0.025	0.040	0.076	0.230	0.025	0.040
fs09000000GET /search/authorities keyword all randomword	233	0.065	0.025	0.047	0.089	0.158	0.270	0.379	0.067	0.158
fs09000000GET /search/authorities keyword any random sentence	233	0.065	0.017	0.092	0.129	0.200	0.315	0.354	0.098	0.199
fs09000000GET /search/authorities keyword any randomword	233	0.065	0.014	0.028	0.062	0.145	0.197	0.228	0.046	0.145
fs09000000GET /search/instances: contributors <> 2random words	233	0.065	0.255	0.274	0.296	0.422	0.752	1.599	0.302	0.420
fs09000000GET /search/instances: contributors = *randomword	233	0.065	0.449	0.481	0.570	0.943	1.357	1.790	0.562	0.942
fs09000000GET /search/instances: contributors = randomword*	233	0.065	0.013	0.041	0.160	0.219	0.242	0.276	0.084	0.218
fs09000000GET /search/instances: contributors == 2random words	233	0.065	0.013	0.020	0.026	0.040	0.069	0.078	0.023	0.039
fs09000000GET /search/instances: contributors all 2random words	233	0.065	0.014	0.017	0.024	0.038	0.136	0.211	0.023	0.038
fs09000000GET /search/instances: contributors all randomword	233	0.065	0.014	0.089	0.179	0.240	0.364	0.441	0.112	0.238
fs09000000GET /search/instances: contributors any 2random words	233	0.065	0.017	0.152	0.196	0.240	0.390	0.622	0.139	0.239
fs09000000GET /search/instances: keyword <> 1word	233	0.065	0.258	0.282	0.324	0.489	0.709	1.219	0.323	0.488
fs09000000GET /search/instances: keyword <> random 2words	233	0.065	0.259	0.281	0.330	0.570	0.725	0.803	0.326	0.569
fs09000000GET /search/instances: keyword = 1word*	233	0.065	0.018	0.170	0.207	0.274	0.456	0.591	0.159	0.273
fs09000000GET /search/instances: keyword = random 2words	233	0.065	0.016	0.051	0.082	0.153	0.299	0.313	0.063	0.153
fs09000000GET /search/instances: keyword == random 2words	233	0.065	0.016	0.028	0.041	0.082	0.122	0.370	0.037	0.082
fs09000000GET /search/instances: keyword all random 2words	233	0.065	0.018	0.059	0.096	0.198	0.397	0.434	0.080	0.197
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	233	0.065	0.020	0.075	0.121	0.194	0.289	0.462	0.093	0.194
fs09000000GET /search/instances: keyword any random 2words	233	0.065	0.049	0.203	0.237	0.343	0.599	1.201	0.220	0.342
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	233	0.065	0.043	0.057	0.172	0.237	0.377	0.453	0.105	0.236
fs09000000GET /search/instances: languages == lang	233	0.065	0.043	0.054	0.148	0.185	0.358	0.461	0.094	0.184
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	233	0.065	0.047	0.070	0.213	0.244	0.494	0.618	0.127	0.243
fs09000000GET /search/instances: subjects <> 1randomword	233	0.065	0.015	0.183	0.219	0.259	0.357	0.697	0.169	0.258
fs09000000GET /search/instances: subjects = (randomword OR randomword)	233	0.065	0.023	0.173	0.216	0.258	0.407	0.453	0.167	0.257
fs09000000GET /search/instances: subjects = *1randomword	233	0.065	0.647	0.743	0.872	1.327	2.303	2.916	0.850	1.325
fs09000000GET /search/instances: subjects = 1randomword*	233	0.065	0.013	0.116	0.158	0.210	0.270	0.289	0.104	0.209
fs09000000GET /search/instances: subjects == 1randomword	233	0.065	0.014	0.054	0.109	0.201	0.245	0.390	0.081	0.199
fs09000000GET /search/instances: subjects all 1randomword	233	0.065	0.015	0.072	0.160	0.229	0.304	0.353	0.099	0.227
fs09000000GET /search/instances: subjects all randomword AND title all randomword	233	0.065	0.015	0.049	0.073	0.130	0.245	0.375	0.059	0.130
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	233	0.065	0.015	0.187	0.236	0.334	0.514	0.745	0.182	0.333
fs09000000GET /search/instances: subjects all randomword OR title all randomword	233	0.065	0.026	0.213	0.256	0.325	0.461	0.528	0.214	0.324
fs09000000GET /search/instances: subjects any 2random words	233	0.065	0.016	0.132	0.176	0.245	0.459	0.802	0.141	0.243
fs09000000GET /search/instances: title <> 3 word sentence	232	0.065	0.260	0.282	0.314	0.499	0.692	0.751	0.320	0.498
fs09000000GET /search/instances: title == 3 word sentence	232	0.065	0.017	0.031	0.050	0.110	0.194	0.230	0.043	0.110
fs09000000GET /search/instances: title all *1randomword	232	0.065	4.602	5.811	8.438	11.234	14.750	15.833	6.886	11.234
fs09000000GET /search/instances: title all 1randomword*	232	0.065	0.017	0.085	0.174	0.236	0.380	0.622	0.117	0.234
fs09000000GET /search/instances: title all 3 word sentence	233	0.065	0.020	0.066	0.115	0.188	0.264	0.298	0.084	0.188
fs09000000GET /search/instances: title any 3 word sentence	233	0.065	0.116	0.313	0.690	1.747	2.429	2.833	0.585	1.745

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.

Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

...

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649439816854&to=1649444031072

CPU Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

1 User with background reindexing

runtime 60 minutes

Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18%

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	175	0.049	0.088	0.221	0.339	0.557	0.685	1.104	0.263	0.557
fs09000000GET /search/authorities keyword <> randomword	175	0.049	0.087	0.158	0.289	0.505	0.592	1.069	0.216	0.505
fs09000000GET /search/authorities keyword = random sentence	175	0.049	0.570	0.917	1.203	1.708	2.337	3.768	1.013	1.708
fs09000000GET /search/authorities keyword = random sentence	175	0.049	0.017	0.021	0.028	0.046	0.067	0.106	0.026	0.046
fs09000000GET /search/authorities keyword = randomword	175	0.049	0.015	0.028	0.054	0.140	0.186	0.342	0.046	0.140
fs09000000GET /search/authorities keyword == random sentence	175	0.049	0.017	0.021	0.029	0.068	0.167	0.210	0.029	0.068
fs09000000GET /search/authorities keyword == randomword	175	0.049	0.015	0.026	0.040	0.150	0.229	0.271	0.042	0.149
fs09000000GET /search/authorities keyword all randomword	175	0.049	0.311	0.657	0.839	1.269	1.490	3.562	0.720	1.269
fs09000000GET /search/authorities keyword all random sentence	175	0.049	0.017	0.026	0.035	0.073	0.137	0.198	0.033	0.073
fs09000000GET /search/authorities keyword all randomword	175	0.049	0.025	0.047	0.120	0.195	0.279	0.652	0.083	0.195
fs09000000GET /search/authorities keyword any random sentence	175	0.049	0.018	0.145	0.190	0.342	0.408	0.425	0.149	0.342
fs09000000GET /search/authorities keyword any randomword	175	0.049	0.014	0.030	0.090	0.208	0.244	0.387	0.062	0.208
fs09000000GET /search/instances: contributors <> 2random words	175	0.049	0.144	0.313	0.387	0.668	0.773	0.969	0.339	0.667
fs09000000GET /search/instances: contributors = *randomword	175	0.049	0.460	0.739	0.940	1.445	2.064	2.245	0.830	1.444
fs09000000GET /search/instances: contributors = randomword*	175	0.049	0.014	0.077	0.213	0.313	0.374	0.407	0.119	0.312
fs09000000GET /search/instances: contributors == 2random words	175	0.049	0.014	0.027	0.036	0.061	0.109	0.208	0.032	0.061
fs09000000GET /search/instances: contributors all 2random words	175	0.049	0.014	0.020	0.028	0.048	0.058	0.072	0.025	0.048
fs09000000GET /search/instances: contributors all randomword	175	0.049	0.015	0.143	0.257	0.325	0.421	0.607	0.159	0.325
fs09000000GET /search/instances: contributors any 2random words	175	0.049	0.019	0.205	0.272	0.381	0.437	0.593	0.198	0.379
fs09000000GET /search/instances: keyword <> 1word	175	0.049	0.144	0.324	0.429	0.653	0.769	1.073	0.352	0.652
fs09000000GET /search/instances: keyword <> random 2words	175	0.049	0.149	0.316	0.418	0.690	1.037	1.177	0.365	0.688
fs09000000GET /search/instances: keyword = 1word*	175	0.049	0.018	0.228	0.267	0.377	0.587	0.799	0.217	0.376
fs09000000GET /search/instances: keyword = random 2words	175	0.049	0.017	0.072	0.123	0.198	0.271	0.421	0.087	0.198
fs09000000GET /search/instances: keyword == random 2words	175	0.049	0.017	0.032	0.074	0.178	0.303	0.345	0.060	0.178
fs09000000GET /search/instances: keyword all random 2words	175	0.049	0.019	0.106	0.156	0.271	0.360	0.435	0.125	0.271
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	175	0.049	0.048	0.169	0.226	0.306	0.492	0.807	0.181	0.306
fs09000000GET /search/instances: keyword any random 2words	175	0.049	0.054	0.282	0.372	0.560	0.796	1.031	0.324	0.559
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	175	0.049	0.045	0.081	0.183	0.284	0.408	0.441	0.125	0.283
fs09000000GET /search/instances: languages == lang	175	0.049	0.044	0.081	0.164	0.299	0.379	0.519	0.120	0.298
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	175	0.049	0.049	0.108	0.234	0.423	0.615	0.676	0.167	0.422
fs09000000GET /search/instances: subjects <> 1randomword	175	0.049	0.016	0.249	0.281	0.380	0.448	0.555	0.228	0.379
fs09000000GET /search/instances: subjects = (randomword OR randomword)	175	0.049	0.024	0.248	0.287	0.363	0.464	0.682	0.242	0.362
fs09000000GET /search/instances: subjects = *1randomword	175	0.049	0.693	1.071	1.305	1.849	2.368	2.855	1.156	1.848
fs09000000GET /search/instances: subjects = 1randomword*	175	0.049	0.014	0.169	0.218	0.283	0.308	0.576	0.144	0.281
fs09000000GET /search/instances: subjects == 1randomword	175	0.049	0.015	0.076	0.159	0.288	0.366	0.565	0.111	0.287
fs09000000GET /search/instances: subjects all 1randomword	175	0.049	0.014	0.112	0.212	0.297	0.397	1.243	0.142	0.296
fs09000000GET /search/instances: subjects all randomword AND title all randomword	175	0.049	0.023	0.088	0.122	0.200	0.297	0.511	0.100	0.200
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	175	0.049	0.023	0.279	0.341	0.445	0.557	0.709	0.265	0.444
fs09000000GET /search/instances: subjects all randomword OR title all randomword	175	0.049	0.068	0.310	0.358	0.521	0.728	0.836	0.326	0.520
fs09000000GET /search/instances: subjects any 2random words	175	0.049	0.020	0.206	0.262	0.345	0.495	0.597	0.201	0.344
fs09000000GET /search/instances: title <> 3 word sentence	175	0.049	0.152	0.347	0.458	0.771	1.131	1.180	0.397	0.770
fs09000000GET /search/instances: title == 3 word sentence	175	0.049	0.017	0.037	0.120	0.204	0.359	0.413	0.079	0.204
fs09000000GET /search/instances: title all *1randomword	175	0.049	5.006	8.103	11.273	18.093	22.114	22.748	9.568	18.093
fs09000000GET /search/instances: title all 1randomword*	175	0.049	0.017	0.113	0.245	0.391	0.504	0.639	0.159	0.390
fs09000000GET /search/instances: title all 3 word sentence	175	0.049	0.039	0.133	0.178	0.274	0.317	0.777	0.148	0.274
fs09000000GET /search/instances: title any 3 word sentence	175	0.049	0.175	0.473	0.995	2.822	3.720	4.922	0.860	2.821

Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.

Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing

Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing

...

http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850

CPU Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.

20 Users without any background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	1508	0.420	0.149	0.723	1.042	1.781	2.589	6.343	0.850	1.781
fs09000000GET /search/authorities keyword <> randomword	1511	0.420	0.156	0.713	1.030	1.840	2.887	5.796	0.851	1.836
fs09000000GET /search/authorities keyword = random sentence	1508	0.420	0.883	1.671	2.048	2.847	4.687	8.326	1.788	2.847
fs09000000GET /search/authorities keyword = random sentence	1508	0.420	0.031	0.460	0.717	1.376	2.219	10.218	0.596	1.376
fs09000000GET /search/authorities keyword = randomword	1511	0.420	0.026	0.449	0.711	1.406	2.594	6.043	0.593	1.404
fs09000000GET /search/authorities keyword == random sentence	1508	0.420	0.068	0.447	0.693	1.278	2.153	5.411	0.558	1.278
fs09000000GET /search/authorities keyword == randomword	1510	0.420	0.016	0.472	0.734	1.561	2.577	7.378	0.614	1.561
fs09000000GET /search/authorities keyword all randomword	1510	0.419	0.545	1.359	1.743	2.545	3.722	6.997	1.468	2.545
fs09000000GET /search/authorities keyword all random sentence	1509	0.419	0.053	0.471	0.722	1.361	2.491	5.385	0.591	1.361
fs09000000GET /search/authorities keyword all randomword	1512	0.420	0.021	0.486	0.771	1.603	2.648	5.197	0.632	1.603
fs09000000GET /search/authorities keyword any random sentence	1508	0.420	0.051	0.507	0.767	1.464	2.614	5.463	0.636	1.464
fs09000000GET /search/authorities keyword any randomword	1512	0.420	0.019	0.485	0.759	1.546	2.559	4.932	0.613	1.546
fs09000000GET /search/instances: contributors <> 2random words	1506	0.420	0.229	0.845	1.202	2.095	3.866	10.696	0.998	2.094
fs09000000GET /search/instances: contributors = *randomword	1506	0.420	0.602	1.360	1.728	2.526	3.411	4.899	1.472	2.526
fs09000000GET /search/instances: contributors = randomword*	1506	0.420	0.064	0.508	0.778	1.548	2.319	6.598	0.642	1.547
fs09000000GET /search/instances: contributors == 2random words	1506	0.420	0.065	0.459	0.702	1.401	2.373	6.619	0.588	1.401
fs09000000GET /search/instances: contributors all 2random words	1506	0.420	0.058	0.472	0.749	1.424	2.618	5.975	0.607	1.424
fs09000000GET /search/instances: contributors all randomword	1508	0.420	0.042	0.550	0.813	1.512	2.460	6.007	0.670	1.512
fs09000000GET /search/instances: contributors any 2random words	1506	0.420	0.090	0.590	0.860	1.558	2.395	11.632	0.726	1.556
fs09000000GET /search/instances: keyword <> 1word	1503	0.419	0.219	0.890	1.219	2.047	2.918	10.988	1.012	2.046
fs09000000GET /search/instances: keyword <> random 2words	1504	0.419	0.215	0.852	1.190	1.924	2.871	6.320	0.975	1.922
fs09000000GET /search/instances: keyword = 1word*	1503	0.419	0.065	0.598	0.867	1.558	2.587	4.883	0.728	1.557
fs09000000GET /search/instances: keyword = random 2words	1504	0.419	0.077	0.472	0.714	1.436	2.553	4.621	0.601	1.434
fs09000000GET /search/instances: keyword == random 2words	1504	0.419	0.079	0.489	0.771	1.417	2.254	6.359	0.614	1.417
fs09000000GET /search/instances: keyword all random 2words	1505	0.419	0.057	0.526	0.770	1.453	2.730	9.947	0.649	1.453
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	1506	0.420	0.066	0.492	0.740	1.497	2.171	4.692	0.619	1.484
fs09000000GET /search/instances: keyword any random 2words	1504	0.419	0.188	0.661	0.941	1.595	2.520	7.017	0.790	1.594
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	1505	0.419	0.103	0.612	0.903	1.685	2.572	8.275	0.746	1.681
fs09000000GET /search/instances: languages == lang	1505	0.419	0.106	0.584	0.876	1.530	2.430	7.004	0.712	1.528
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	1506	0.419	0.099	0.638	0.929	1.727	2.791	7.607	0.778	1.720
fs09000000GET /search/instances: subjects <> 1randomword	1498	0.418	0.082	0.589	0.867	1.665	3.147	14.882	0.744	1.664
fs09000000GET /search/instances: subjects = (randomword OR randomword)	1508	0.420	0.107	0.591	0.861	1.535	2.498	5.127	0.719	1.534
fs09000000GET /search/instances: subjects = *1randomword	1500	0.418	0.827	1.796	2.211	3.160	4.472	7.100	1.915	3.160
fs09000000GET /search/instances: subjects = 1randomword*	1502	0.419	0.052	0.544	0.804	1.495	2.269	7.514	0.660	1.495
fs09000000GET /search/instances: subjects == 1randomword	1499	0.418	0.057	0.554	0.840	1.696	2.676	4.595	0.694	1.695
fs09000000GET /search/instances: subjects all 1randomword	1502	0.419	0.070	0.547	0.800	1.416	2.367	4.632	0.658	1.415
fs09000000GET /search/instances: subjects all randomword AND title all randomword	1508	0.420	0.068	0.489	0.767	1.456	2.203	11.435	0.619	1.456
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	1508	0.420	0.096	0.619	0.916	1.653	2.430	6.527	0.748	1.649
fs09000000GET /search/instances: subjects all randomword OR title all randomword	1508	0.420	0.152	0.622	0.896	1.652	2.654	7.750	0.758	1.650
fs09000000GET /search/instances: subjects any 2random words	1499	0.418	0.088	0.582	0.873	1.559	2.656	8.047	0.724	1.557
fs09000000GET /search/instances: title <> 3 word sentence	1497	0.418	0.266	0.888	1.229	2.040	4.183	7.922	1.028	2.038
fs09000000GET /search/instances: title == 3 word sentence	1497	0.418	0.079	0.499	0.782	1.551	2.437	8.389	0.640	1.551
fs09000000GET /search/instances: title all *1randomword	1497	0.417	5.253	10.627	13.455	19.914	22.201	28.924	11.478	19.914
fs09000000GET /search/instances: title all 1randomword*	1497	0.418	0.064	0.599	0.907	1.706	3.154	7.831	0.764	1.704
fs09000000GET /search/instances: title all 3 word sentence	1498	0.418	0.072	0.482	0.756	1.504	2.588	6.353	0.630	1.504
fs09000000GET /search/instances: title any 3 word sentence	1498	0.418	0.270	1.008	1.705	3.755	5.009	14.345	1.396	3.754

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~28 seconds For all other queries it is closer to 10 seconds or less.

Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

Response time distribution

Out of ~69K requests, 31K requests had a response time > 700 milliseconds All these 31K requests were wildcard queries

CPU Utilization

mod-search's CPU was reasonable at around 30% throughout the test run which is 20% more than 1 user.

...

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649944107574&to=1649948967673

20 Users with background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	1297	0.361	0.202	0.823	1.113	1.955	2.636	3.544	0.936	1.955
fs09000000GET /search/authorities keyword <> randomword	1299	0.361	0.110	0.794	1.136	2.123	3.487	9.501	0.968	2.123
fs09000000GET /search/authorities keyword = random sentence	1295	0.360	0.935	1.935	2.400	3.303	4.113	11.918	2.045	3.303
fs09000000GET /search/authorities keyword = random sentence	1295	0.360	0.102	0.551	0.815	1.523	2.523	10.937	0.680	1.523
fs09000000GET /search/authorities keyword = randomword	1300	0.361	0.016	0.551	0.827	1.679	2.680	12.588	0.707	1.679
fs09000000GET /search/authorities keyword == random sentence	1297	0.361	0.094	0.565	0.869	1.626	2.534	8.289	0.711	1.626
fs09000000GET /search/authorities keyword == randomword	1298	0.361	0.021	0.556	0.839	1.608	2.582	8.349	0.701	1.608
fs09000000GET /search/authorities keyword all randomword	1298	0.361	0.564	1.520	1.997	3.083	4.290	10.092	1.691	3.083
fs09000000GET /search/authorities keyword all random sentence	1297	0.361	0.105	0.549	0.829	1.657	2.655	8.973	0.688	1.657
fs09000000GET /search/authorities keyword all randomword	1300	0.361	0.017	0.564	0.830	1.617	2.824	9.527	0.714	1.617
fs09000000GET /search/authorities keyword any random sentence	1297	0.361	0.147	0.604	0.868	1.763	2.818	9.430	0.751	1.763
fs09000000GET /search/authorities keyword any randomword	1300	0.361	0.018	0.567	0.848	1.645	3.012	8.022	0.721	1.645
fs09000000GET /search/instances: contributors <> 2random words	1292	0.361	0.246	0.935	1.273	2.185	3.219	7.719	1.072	2.184
fs09000000GET /search/instances: contributors = *randomword	1293	0.360	0.672	1.611	2.000	2.919	4.005	13.528	1.764	2.919
fs09000000GET /search/instances: contributors = randomword*	1293	0.360	0.094	0.615	0.948	1.813	3.178	8.855	0.791	1.813
fs09000000GET /search/instances: contributors == 2random words	1292	0.361	0.118	0.542	0.809	1.567	2.535	9.101	0.675	1.567
fs09000000GET /search/instances: contributors all 2random words	1292	0.361	0.092	0.566	0.813	1.600	2.309	13.326	0.685	1.600
fs09000000GET /search/instances: contributors all randomword	1293	0.360	0.139	0.665	0.984	1.889	2.997	11.820	0.829	1.887
fs09000000GET /search/instances: contributors any 2random words	1292	0.361	0.152	0.692	0.977	1.842	3.031	8.169	0.844	1.841
fs09000000GET /search/instances: keyword <> 1word	1286	0.359	0.280	0.969	1.312	2.150	3.066	5.673	1.100	2.149
fs09000000GET /search/instances: keyword <> random 2words	1287	0.360	0.303	0.961	1.303	2.139	3.006	12.240	1.106	2.138
fs09000000GET /search/instances: keyword = 1word*	1287	0.360	0.168	0.738	1.034	1.908	2.872	9.941	0.885	1.907
fs09000000GET /search/instances: keyword = random 2words	1289	0.360	0.115	0.572	0.884	1.639	2.549	5.710	0.714	1.639
fs09000000GET /search/instances: keyword == random 2words	1287	0.360	0.131	0.570	0.870	1.759	2.572	8.373	0.725	1.759
fs09000000GET /search/instances: keyword all random 2words	1290	0.360	0.117	0.606	0.916	1.732	2.678	11.065	0.772	1.731
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	1292	0.361	0.108	0.592	0.875	1.633	2.548	8.574	0.733	1.633
fs09000000GET /search/instances: keyword any random 2words	1289	0.360	0.269	0.775	1.113	1.960	2.838	8.873	0.931	1.959
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	1292	0.361	0.188	0.741	1.035	1.821	2.858	11.866	0.868	1.819
fs09000000GET /search/instances: languages == lang	1292	0.361	0.175	0.686	0.983	1.827	2.765	5.447	0.823	1.826
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	1292	0.361	0.164	0.778	1.089	1.927	3.108	12.475	0.922	1.926
fs09000000GET /search/instances: subjects <> 1randomword	1285	0.360	0.148	0.697	0.984	1.728	2.618	9.452	0.839	1.727
fs09000000GET /search/instances: subjects = (randomword OR randomword)	1294	0.360	0.200	0.735	1.032	1.852	3.152	11.812	0.892	1.847
fs09000000GET /search/instances: subjects = *1randomword	1285	0.360	0.923	2.121	2.557	3.713	5.281	8.462	2.252	3.712
fs09000000GET /search/instances: subjects = 1randomword*	1285	0.360	0.130	0.641	0.945	1.668	2.945	9.114	0.787	1.667
fs09000000GET /search/instances: subjects == 1randomword	1285	0.360	0.107	0.612	0.918	1.725	2.919	11.933	0.775	1.724
fs09000000GET /search/instances: subjects all 1randomword	1286	0.359	0.152	0.668	0.984	1.815	2.787	12.340	0.815	1.813
fs09000000GET /search/instances: subjects all randomword AND title all randomword	1294	0.360	0.118	0.605	0.873	1.727	2.714	6.724	0.739	1.727
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	1294	0.360	0.123	0.738	1.064	1.852	2.876	5.605	0.882	1.851
fs09000000GET /search/instances: subjects all randomword OR title all randomword	1294	0.360	0.206	0.747	1.051	1.986	3.539	11.722	0.924	1.985
fs09000000GET /search/instances: subjects any 2random words	1285	0.360	0.131	0.691	0.967	1.839	2.728	9.587	0.839	1.837
fs09000000GET /search/instances: title <> 3 word sentence	1285	0.360	0.266	0.959	1.313	2.103	3.439	16.493	1.132	2.102
fs09000000GET /search/instances: title == 3 word sentence	1285	0.360	0.078	0.593	0.890	1.796	4.453	16.146	0.820	1.796
fs09000000GET /search/instances: title all *1randomword	1284	0.359	5.481	12.403	15.789	22.022	25.269	33.003	13.309	22.022
fs09000000GET /search/instances: title all 1randomword*	1284	0.360	0.120	0.682	0.981	1.988	3.632	12.476	0.865	1.987
fs09000000GET /search/instances: title all 3 word sentence	1285	0.360	0.097	0.590	0.864	1.766	3.475	12.152	0.779	1.766
fs09000000GET /search/instances: title any 3 word sentence	1285	0.360	0.326	1.109	1.942	4.373	6.041	12.549	1.612	4.371

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~30 seconds For all other queries it is closer to 10 seconds or less.

Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

Response time distribution

Out of ~69K requests, 32K requests had a response time > 700 milliseconds All these 32K requests were wildcard queries

CPU Utilization

CPU spikes linearly 40% as we start background reindexing. When we start the JMeter script run, the CPU continues to move upwards and reaches a max of 70%. Okapi also consumes resources but it is less than 40% which reasonable.

...

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1650032755868&to=1650036953158

JIRA tickets created as part of this investigation

Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MSEARCH-331
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MSEARCH-337

Additional information

Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000

Artifact stored http://carrier-io.int.folio.ebsco.com/artifacts?q=mod-search - mod-search.zip

...

Versions Compared

Old Version 42

New Version 43

Key

Overview

High-Level Summary

Test Runs

1 User without any background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

1 User with background reindexing

Response Time Over Time

Throughput

Response Time distribution

CPU Utilization

20 Users without any background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

20 Users with background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

JIRA tickets created as part of this investigation

Additional information

Page Comparison

Versions Compared

Old Version 42

New Version 43

Key

Overview

High-Level Summary

Test Runs

1 User without any background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

1 User with background reindexing

Response Time Over Time

Throughput

Response Time distribution

CPU Utilization

20 Users without any background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

20 Users with background reindexing

Response time over time

Throughput

Response time distribution

CPU Utilization

JIRA tickets created as part of this investigation

Additional information