Overview

In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - PERF-225 - Getting issue details... STATUS

We tested with 1, 20 virtual users against 10 million Inventory records.

Test plan:

Run reindex for tenant A.
Run test with n users for tenant B

Backend:

mod-search-1.6.4
mod-inventory-18.1.4
mod-inventory-storage-23.0.2
okapi-4.13.0

Frontend:

folio_inventory-9.0.1

Environment:

7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
80 FOLIO back-end modules deployed in 166 ECS services
3 okapi ECS services
7 m5.xlarge EC2 instances
1 writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

authority key search

boolean search

contributors search

filter search

keyword search

subjects search
title search
wildcard search
wildcard prefix search

mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

Query	1 user - Baseline (avg)	1 user - Under the reindexing load (avg)	20 user - Baseline (avg)	20 user - Under the reindexing load (avg)
Regular	336 ms	448 ms	728 ms	0.850 ms
Wildcard	1.42 seconds	1.63 seconds	2.32 seconds	2.71 seconds (2 failed requests for /search/instances: title all *1randomword)

*Note that "Regular" queries are queries that don't have the wild card character ( * ).

Show total requests processed increases as we increase the number of users concurrently.

	1 user - Baseline (avg) ms	1 user - Under the load (avg) ms	20 user - Under the load (avg) ms	20 user - Under the load (avg) ms
Total requests processed in 60 minutes test run	10715	8051	69244	59420

Comparison with Kiwi

	1 user		1 user reindexing		20 users		20 users reindexing
	Kiwi(ms)	Lotus(ms)	Kiwi(ms)	Lotus(ms)	Kiwi(ms)	Lotus(ms)	Kiwi(ms)	Lotus(ms)
Regular	648	336	569	448	3190	728	7009	850
Wildcard queries	660	268	561	159	4185	698	11160	832
Wildcard prefix queries	13441	1740	11569	2657	17424	3624	22009	4212

	1 user		1 user reindexing		20 users		20 users reindexing
	Percentage	Times better than Kiwi	Percentage	Times better than Kiwi	Percentage	Times better than Kiwi	Percentage	Times better than Kiwi
Regular Delta	48.15%	1	21.27%	1	77.18%	4	87.87%	8
Wildcard queries Delta	59.39%	2	71.66%	3	83.32%	5	92.54%	13
Wildcard prefix queries Delta	87.05%	7	77.03%	4	79.20%	4	80.86%	5

	CPU		Memory		Number of Tasks
	Kiwi	Lotus	Kiwi	Lotus	Kiwi	Lotus
mod-search	256	256	536	1440	2	1
okapi	256	128	536	544	3	3
mod-authtoken	128	128	360	322	2	2
mod-permissions	128	128	536	537	2	2
Elastic Search	No differences

Test Runs

1 User without any background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	233	0.065	0.250	0.264	0.298	0.520	0.721	0.973	0.302	0.520
fs09000000GET /search/authorities keyword <> randomword	233	0.065	0.246	0.261	0.289	0.548	0.810	1.771	0.312	0.548
fs09000000GET /search/authorities keyword = random sentence	233	0.065	0.567	0.710	0.962	1.681	2.240	2.582	0.874	1.681
fs09000000GET /search/authorities keyword = random sentence	233	0.065	0.017	0.021	0.023	0.034	0.048	0.074	0.023	0.034
fs09000000GET /search/authorities keyword = randomword	233	0.065	0.014	0.024	0.056	0.135	0.204	0.449	0.044	0.135
fs09000000GET /search/authorities keyword == random sentence	233	0.065	0.017	0.020	0.022	0.035	0.055	0.101	0.022	0.033
fs09000000GET /search/authorities keyword == randomword	233	0.065	0.015	0.022	0.034	0.089	0.164	0.241	0.034	0.089
fs09000000GET /search/authorities keyword all randomword	233	0.065	0.302	0.554	0.824	1.198	1.675	2.013	0.644	1.198
fs09000000GET /search/authorities keyword all random sentence	233	0.065	0.017	0.021	0.025	0.040	0.076	0.230	0.025	0.040
fs09000000GET /search/authorities keyword all randomword	233	0.065	0.025	0.047	0.089	0.158	0.270	0.379	0.067	0.158
fs09000000GET /search/authorities keyword any random sentence	233	0.065	0.017	0.092	0.129	0.200	0.315	0.354	0.098	0.199
fs09000000GET /search/authorities keyword any randomword	233	0.065	0.014	0.028	0.062	0.145	0.197	0.228	0.046	0.145
fs09000000GET /search/instances: contributors <> 2random words	233	0.065	0.255	0.274	0.296	0.422	0.752	1.599	0.302	0.420
fs09000000GET /search/instances: contributors = *randomword	233	0.065	0.449	0.481	0.570	0.943	1.357	1.790	0.562	0.942
fs09000000GET /search/instances: contributors = randomword*	233	0.065	0.013	0.041	0.160	0.219	0.242	0.276	0.084	0.218
fs09000000GET /search/instances: contributors == 2random words	233	0.065	0.013	0.020	0.026	0.040	0.069	0.078	0.023	0.039
fs09000000GET /search/instances: contributors all 2random words	233	0.065	0.014	0.017	0.024	0.038	0.136	0.211	0.023	0.038
fs09000000GET /search/instances: contributors all randomword	233	0.065	0.014	0.089	0.179	0.240	0.364	0.441	0.112	0.238
fs09000000GET /search/instances: contributors any 2random words	233	0.065	0.017	0.152	0.196	0.240	0.390	0.622	0.139	0.239
fs09000000GET /search/instances: keyword <> 1word	233	0.065	0.258	0.282	0.324	0.489	0.709	1.219	0.323	0.488
fs09000000GET /search/instances: keyword <> random 2words	233	0.065	0.259	0.281	0.330	0.570	0.725	0.803	0.326	0.569
fs09000000GET /search/instances: keyword = 1word*	233	0.065	0.018	0.170	0.207	0.274	0.456	0.591	0.159	0.273
fs09000000GET /search/instances: keyword = random 2words	233	0.065	0.016	0.051	0.082	0.153	0.299	0.313	0.063	0.153
fs09000000GET /search/instances: keyword == random 2words	233	0.065	0.016	0.028	0.041	0.082	0.122	0.370	0.037	0.082
fs09000000GET /search/instances: keyword all random 2words	233	0.065	0.018	0.059	0.096	0.198	0.397	0.434	0.080	0.197
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	233	0.065	0.020	0.075	0.121	0.194	0.289	0.462	0.093	0.194
fs09000000GET /search/instances: keyword any random 2words	233	0.065	0.049	0.203	0.237	0.343	0.599	1.201	0.220	0.342
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	233	0.065	0.043	0.057	0.172	0.237	0.377	0.453	0.105	0.236
fs09000000GET /search/instances: languages == lang	233	0.065	0.043	0.054	0.148	0.185	0.358	0.461	0.094	0.184
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	233	0.065	0.047	0.070	0.213	0.244	0.494	0.618	0.127	0.243
fs09000000GET /search/instances: subjects <> 1randomword	233	0.065	0.015	0.183	0.219	0.259	0.357	0.697	0.169	0.258
fs09000000GET /search/instances: subjects = (randomword OR randomword)	233	0.065	0.023	0.173	0.216	0.258	0.407	0.453	0.167	0.257
fs09000000GET /search/instances: subjects = *1randomword	233	0.065	0.647	0.743	0.872	1.327	2.303	2.916	0.850	1.325
fs09000000GET /search/instances: subjects = 1randomword*	233	0.065	0.013	0.116	0.158	0.210	0.270	0.289	0.104	0.209
fs09000000GET /search/instances: subjects == 1randomword	233	0.065	0.014	0.054	0.109	0.201	0.245	0.390	0.081	0.199
fs09000000GET /search/instances: subjects all 1randomword	233	0.065	0.015	0.072	0.160	0.229	0.304	0.353	0.099	0.227
fs09000000GET /search/instances: subjects all randomword AND title all randomword	233	0.065	0.015	0.049	0.073	0.130	0.245	0.375	0.059	0.130
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	233	0.065	0.015	0.187	0.236	0.334	0.514	0.745	0.182	0.333
fs09000000GET /search/instances: subjects all randomword OR title all randomword	233	0.065	0.026	0.213	0.256	0.325	0.461	0.528	0.214	0.324
fs09000000GET /search/instances: subjects any 2random words	233	0.065	0.016	0.132	0.176	0.245	0.459	0.802	0.141	0.243
fs09000000GET /search/instances: title <> 3 word sentence	232	0.065	0.260	0.282	0.314	0.499	0.692	0.751	0.320	0.498
fs09000000GET /search/instances: title == 3 word sentence	232	0.065	0.017	0.031	0.050	0.110	0.194	0.230	0.043	0.110
fs09000000GET /search/instances: title all *1randomword	232	0.065	4.602	5.811	8.438	11.234	14.750	15.833	6.886	11.234
fs09000000GET /search/instances: title all 1randomword*	232	0.065	0.017	0.085	0.174	0.236	0.380	0.622	0.117	0.234
fs09000000GET /search/instances: title all 3 word sentence	233	0.065	0.020	0.066	0.115	0.188	0.264	0.298	0.084	0.188
fs09000000GET /search/instances: title any 3 word sentence	233	0.065	0.116	0.313	0.690	1.747	2.429	2.833	0.585	1.745

Wildcard queries	Avg (seconds)	Error
fs09000000GET /search/instances: contributors = randomword*	0.084	0.0%
fs09000000GET /search/instances: keyword = 1word*	0.159	0.0%
fs09000000GET /search/instances: subjects = 1randomword*	0.104	0.0%
fs09000000GET /search/instances: title all 1randomword*	0.117	0.0%
	0.2676

Wildcard prefix queries	Avg (seconds)	Error
fs09000000GET /search/authorities keyword = random sentence	0.874	0.0%
fs09000000GET /search/authorities keyword all randomword	0.644	0.0%
fs09000000GET /search/instances: contributors = *randomword	0.562	0.0%
fs09000000GET /search/authorities keyword all randomword	0.644	0.0%
fs09000000GET /search/instances: subjects = *1randomword	0.85	0.0%
fs09000000GET /search/instances: title all *1randomword	6.886	0.0%
	1.74

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.

Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649439816854&to=1649444031072

CPU and Memory Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

mod-search memory is stable around 40% throughtout the test run

1 User with background reindexing

runtime 60 minutes

Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18%

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	175	0.049	0.088	0.221	0.339	0.557	0.685	1.104	0.263	0.557
fs09000000GET /search/authorities keyword <> randomword	175	0.049	0.087	0.158	0.289	0.505	0.592	1.069	0.216	0.505
fs09000000GET /search/authorities keyword = random sentence	175	0.049	0.570	0.917	1.203	1.708	2.337	3.768	1.013	1.708
fs09000000GET /search/authorities keyword = random sentence	175	0.049	0.017	0.021	0.028	0.046	0.067	0.106	0.026	0.046
fs09000000GET /search/authorities keyword = randomword	175	0.049	0.015	0.028	0.054	0.140	0.186	0.342	0.046	0.140
fs09000000GET /search/authorities keyword == random sentence	175	0.049	0.017	0.021	0.029	0.068	0.167	0.210	0.029	0.068
fs09000000GET /search/authorities keyword == randomword	175	0.049	0.015	0.026	0.040	0.150	0.229	0.271	0.042	0.149
fs09000000GET /search/authorities keyword all randomword	175	0.049	0.311	0.657	0.839	1.269	1.490	3.562	0.720	1.269
fs09000000GET /search/authorities keyword all random sentence	175	0.049	0.017	0.026	0.035	0.073	0.137	0.198	0.033	0.073
fs09000000GET /search/authorities keyword all randomword	175	0.049	0.025	0.047	0.120	0.195	0.279	0.652	0.083	0.195
fs09000000GET /search/authorities keyword any random sentence	175	0.049	0.018	0.145	0.190	0.342	0.408	0.425	0.149	0.342
fs09000000GET /search/authorities keyword any randomword	175	0.049	0.014	0.030	0.090	0.208	0.244	0.387	0.062	0.208
fs09000000GET /search/instances: contributors <> 2random words	175	0.049	0.144	0.313	0.387	0.668	0.773	0.969	0.339	0.667
fs09000000GET /search/instances: contributors = *randomword	175	0.049	0.460	0.739	0.940	1.445	2.064	2.245	0.830	1.444
fs09000000GET /search/instances: contributors = randomword*	175	0.049	0.014	0.077	0.213	0.313	0.374	0.407	0.119	0.312
fs09000000GET /search/instances: contributors == 2random words	175	0.049	0.014	0.027	0.036	0.061	0.109	0.208	0.032	0.061
fs09000000GET /search/instances: contributors all 2random words	175	0.049	0.014	0.020	0.028	0.048	0.058	0.072	0.025	0.048
fs09000000GET /search/instances: contributors all randomword	175	0.049	0.015	0.143	0.257	0.325	0.421	0.607	0.159	0.325
fs09000000GET /search/instances: contributors any 2random words	175	0.049	0.019	0.205	0.272	0.381	0.437	0.593	0.198	0.379
fs09000000GET /search/instances: keyword <> 1word	175	0.049	0.144	0.324	0.429	0.653	0.769	1.073	0.352	0.652
fs09000000GET /search/instances: keyword <> random 2words	175	0.049	0.149	0.316	0.418	0.690	1.037	1.177	0.365	0.688
fs09000000GET /search/instances: keyword = 1word*	175	0.049	0.018	0.228	0.267	0.377	0.587	0.799	0.217	0.376
fs09000000GET /search/instances: keyword = random 2words	175	0.049	0.017	0.072	0.123	0.198	0.271	0.421	0.087	0.198
fs09000000GET /search/instances: keyword == random 2words	175	0.049	0.017	0.032	0.074	0.178	0.303	0.345	0.060	0.178
fs09000000GET /search/instances: keyword all random 2words	175	0.049	0.019	0.106	0.156	0.271	0.360	0.435	0.125	0.271
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	175	0.049	0.048	0.169	0.226	0.306	0.492	0.807	0.181	0.306
fs09000000GET /search/instances: keyword any random 2words	175	0.049	0.054	0.282	0.372	0.560	0.796	1.031	0.324	0.559
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	175	0.049	0.045	0.081	0.183	0.284	0.408	0.441	0.125	0.283
fs09000000GET /search/instances: languages == lang	175	0.049	0.044	0.081	0.164	0.299	0.379	0.519	0.120	0.298
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	175	0.049	0.049	0.108	0.234	0.423	0.615	0.676	0.167	0.422
fs09000000GET /search/instances: subjects <> 1randomword	175	0.049	0.016	0.249	0.281	0.380	0.448	0.555	0.228	0.379
fs09000000GET /search/instances: subjects = (randomword OR randomword)	175	0.049	0.024	0.248	0.287	0.363	0.464	0.682	0.242	0.362
fs09000000GET /search/instances: subjects = *1randomword	175	0.049	0.693	1.071	1.305	1.849	2.368	2.855	1.156	1.848
fs09000000GET /search/instances: subjects = 1randomword*	175	0.049	0.014	0.169	0.218	0.283	0.308	0.576	0.144	0.281
fs09000000GET /search/instances: subjects == 1randomword	175	0.049	0.015	0.076	0.159	0.288	0.366	0.565	0.111	0.287
fs09000000GET /search/instances: subjects all 1randomword	175	0.049	0.014	0.112	0.212	0.297	0.397	1.243	0.142	0.296
fs09000000GET /search/instances: subjects all randomword AND title all randomword	175	0.049	0.023	0.088	0.122	0.200	0.297	0.511	0.100	0.200
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	175	0.049	0.023	0.279	0.341	0.445	0.557	0.709	0.265	0.444
fs09000000GET /search/instances: subjects all randomword OR title all randomword	175	0.049	0.068	0.310	0.358	0.521	0.728	0.836	0.326	0.520
fs09000000GET /search/instances: subjects any 2random words	175	0.049	0.020	0.206	0.262	0.345	0.495	0.597	0.201	0.344
fs09000000GET /search/instances: title <> 3 word sentence	175	0.049	0.152	0.347	0.458	0.771	1.131	1.180	0.397	0.770
fs09000000GET /search/instances: title == 3 word sentence	175	0.049	0.017	0.037	0.120	0.204	0.359	0.413	0.079	0.204
fs09000000GET /search/instances: title all *1randomword	175	0.049	5.006	8.103	11.273	18.093	22.114	22.748	9.568	18.093
fs09000000GET /search/instances: title all 1randomword*	175	0.049	0.017	0.113	0.245	0.391	0.504	0.639	0.159	0.390
fs09000000GET /search/instances: title all 3 word sentence	175	0.049	0.039	0.133	0.178	0.274	0.317	0.777	0.148	0.274
fs09000000GET /search/instances: title any 3 word sentence	175	0.049	0.175	0.473	0.995	2.822	3.720	4.922	0.860	2.821

Wildcard queries	Avg (seconds)	Error
fs09000000GET /search/instances: contributors = randomword*	0.119	0.0%
fs09000000GET /search/instances: keyword = 1word*	0.217	0.0%
fs09000000GET /search/instances: subjects = 1randomword*	0.144	0.0%
fs09000000GET /search/instances: title all 1randomword*	0.159	0.0%
	0.159

Wildcard prefix queries	Avg (seconds)	Error
fs09000000GET /search/authorities keyword = random sentence	1.013	0.0%
fs09000000GET /search/authorities keyword all randomword	0.720	0.0%
fs09000000GET /search/instances: contributors = *randomword	0.830	0.0%
fs09000000GET /search/instances: subjects = *1randomword	1.156	0.0%
fs09000000GET /search/instances: title all *1randomword	9.568	0.0%
	2.657

Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.

Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing

Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing

http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850

CPU and Memory Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.

mod-search memory is stable at around 65% throughout the test run and also during background reindexing

20 Users without any background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	1508	0.420	0.149	0.723	1.042	1.781	2.589	6.343	0.850	1.781
fs09000000GET /search/authorities keyword <> randomword	1511	0.420	0.156	0.713	1.030	1.840	2.887	5.796	0.851	1.836
fs09000000GET /search/authorities keyword = random sentence	1508	0.420	0.883	1.671	2.048	2.847	4.687	8.326	1.788	2.847
fs09000000GET /search/authorities keyword = random sentence	1508	0.420	0.031	0.460	0.717	1.376	2.219	10.218	0.596	1.376
fs09000000GET /search/authorities keyword = randomword	1511	0.420	0.026	0.449	0.711	1.406	2.594	6.043	0.593	1.404
fs09000000GET /search/authorities keyword == random sentence	1508	0.420	0.068	0.447	0.693	1.278	2.153	5.411	0.558	1.278
fs09000000GET /search/authorities keyword == randomword	1510	0.420	0.016	0.472	0.734	1.561	2.577	7.378	0.614	1.561
fs09000000GET /search/authorities keyword all randomword	1510	0.419	0.545	1.359	1.743	2.545	3.722	6.997	1.468	2.545
fs09000000GET /search/authorities keyword all random sentence	1509	0.419	0.053	0.471	0.722	1.361	2.491	5.385	0.591	1.361
fs09000000GET /search/authorities keyword all randomword	1512	0.420	0.021	0.486	0.771	1.603	2.648	5.197	0.632	1.603
fs09000000GET /search/authorities keyword any random sentence	1508	0.420	0.051	0.507	0.767	1.464	2.614	5.463	0.636	1.464
fs09000000GET /search/authorities keyword any randomword	1512	0.420	0.019	0.485	0.759	1.546	2.559	4.932	0.613	1.546
fs09000000GET /search/instances: contributors <> 2random words	1506	0.420	0.229	0.845	1.202	2.095	3.866	10.696	0.998	2.094
fs09000000GET /search/instances: contributors = *randomword	1506	0.420	0.602	1.360	1.728	2.526	3.411	4.899	1.472	2.526
fs09000000GET /search/instances: contributors = randomword*	1506	0.420	0.064	0.508	0.778	1.548	2.319	6.598	0.642	1.547
fs09000000GET /search/instances: contributors == 2random words	1506	0.420	0.065	0.459	0.702	1.401	2.373	6.619	0.588	1.401
fs09000000GET /search/instances: contributors all 2random words	1506	0.420	0.058	0.472	0.749	1.424	2.618	5.975	0.607	1.424
fs09000000GET /search/instances: contributors all randomword	1508	0.420	0.042	0.550	0.813	1.512	2.460	6.007	0.670	1.512
fs09000000GET /search/instances: contributors any 2random words	1506	0.420	0.090	0.590	0.860	1.558	2.395	11.632	0.726	1.556
fs09000000GET /search/instances: keyword <> 1word	1503	0.419	0.219	0.890	1.219	2.047	2.918	10.988	1.012	2.046
fs09000000GET /search/instances: keyword <> random 2words	1504	0.419	0.215	0.852	1.190	1.924	2.871	6.320	0.975	1.922
fs09000000GET /search/instances: keyword = 1word*	1503	0.419	0.065	0.598	0.867	1.558	2.587	4.883	0.728	1.557
fs09000000GET /search/instances: keyword = random 2words	1504	0.419	0.077	0.472	0.714	1.436	2.553	4.621	0.601	1.434
fs09000000GET /search/instances: keyword == random 2words	1504	0.419	0.079	0.489	0.771	1.417	2.254	6.359	0.614	1.417
fs09000000GET /search/instances: keyword all random 2words	1505	0.419	0.057	0.526	0.770	1.453	2.730	9.947	0.649	1.453
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	1506	0.420	0.066	0.492	0.740	1.497	2.171	4.692	0.619	1.484
fs09000000GET /search/instances: keyword any random 2words	1504	0.419	0.188	0.661	0.941	1.595	2.520	7.017	0.790	1.594
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	1505	0.419	0.103	0.612	0.903	1.685	2.572	8.275	0.746	1.681
fs09000000GET /search/instances: languages == lang	1505	0.419	0.106	0.584	0.876	1.530	2.430	7.004	0.712	1.528
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	1506	0.419	0.099	0.638	0.929	1.727	2.791	7.607	0.778	1.720
fs09000000GET /search/instances: subjects <> 1randomword	1498	0.418	0.082	0.589	0.867	1.665	3.147	14.882	0.744	1.664
fs09000000GET /search/instances: subjects = (randomword OR randomword)	1508	0.420	0.107	0.591	0.861	1.535	2.498	5.127	0.719	1.534
fs09000000GET /search/instances: subjects = *1randomword	1500	0.418	0.827	1.796	2.211	3.160	4.472	7.100	1.915	3.160
fs09000000GET /search/instances: subjects = 1randomword*	1502	0.419	0.052	0.544	0.804	1.495	2.269	7.514	0.660	1.495
fs09000000GET /search/instances: subjects == 1randomword	1499	0.418	0.057	0.554	0.840	1.696	2.676	4.595	0.694	1.695
fs09000000GET /search/instances: subjects all 1randomword	1502	0.419	0.070	0.547	0.800	1.416	2.367	4.632	0.658	1.415
fs09000000GET /search/instances: subjects all randomword AND title all randomword	1508	0.420	0.068	0.489	0.767	1.456	2.203	11.435	0.619	1.456
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	1508	0.420	0.096	0.619	0.916	1.653	2.430	6.527	0.748	1.649
fs09000000GET /search/instances: subjects all randomword OR title all randomword	1508	0.420	0.152	0.622	0.896	1.652	2.654	7.750	0.758	1.650
fs09000000GET /search/instances: subjects any 2random words	1499	0.418	0.088	0.582	0.873	1.559	2.656	8.047	0.724	1.557
fs09000000GET /search/instances: title <> 3 word sentence	1497	0.418	0.266	0.888	1.229	2.040	4.183	7.922	1.028	2.038
fs09000000GET /search/instances: title == 3 word sentence	1497	0.418	0.079	0.499	0.782	1.551	2.437	8.389	0.640	1.551
fs09000000GET /search/instances: title all *1randomword	1497	0.417	5.253	10.627	13.455	19.914	22.201	28.924	11.478	19.914
fs09000000GET /search/instances: title all 1randomword*	1497	0.418	0.064	0.599	0.907	1.706	3.154	7.831	0.764	1.704
fs09000000GET /search/instances: title all 3 word sentence	1498	0.418	0.072	0.482	0.756	1.504	2.588	6.353	0.630	1.504
fs09000000GET /search/instances: title any 3 word sentence	1498	0.418	0.270	1.008	1.705	3.755	5.009	14.345	1.396	3.754

Wildcard queries	Avg (seconds)	Error
fs09000000GET /search/instances: contributors = randomword*	0.642	0.0%
fs09000000GET /search/instances: keyword = 1word*	0.728	0.0%
fs09000000GET /search/instances: subjects = 1randomword*	0.660	0.0%
fs09000000GET /search/instances: title all 1randomword*	0.764	0.0%
	0.698

Wildcard prefix queries	Avg (seconds)	Error
fs09000000GET /search/authorities keyword = random sentence	1.788	0.0%
fs09000000GET /search/authorities keyword all randomword	1.468	0.0%
fs09000000GET /search/instances: contributors = *randomword	1.472	0.0%
fs09000000GET /search/instances: subjects = *1randomword	1.915	0.0%
fs09000000GET /search/instances: title all *1randomword	11.478	0.0%
	3.624

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~28 seconds For all other queries it is closer to 10 seconds or less.

Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

Response time distribution

Out of ~69K requests, 31K requests had a response time > 700 milliseconds All these 31K requests were wildcard queries

CPU and Memory Utilization

mod-search's CPU was reasonable at around 30% throughout the test run which is 20% more than 1 user.

Memory is stable at around 65% throughout the test run

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649944107574&to=1649948967673

20 Users with background reindexing

runtime 60 minutes

Requests	Total	Req/s	Min	50pct	75pct	95pct	99pct	Max	Avg	Latency
fs09000000GET /search/authorities keyword <> random sentence	1297	0.361	0.202	0.823	1.113	1.955	2.636	3.544	0.936	1.955
fs09000000GET /search/authorities keyword <> randomword	1299	0.361	0.110	0.794	1.136	2.123	3.487	9.501	0.968	2.123
fs09000000GET /search/authorities keyword = random sentence	1295	0.360	0.935	1.935	2.400	3.303	4.113	11.918	2.045	3.303
fs09000000GET /search/authorities keyword = random sentence	1295	0.360	0.102	0.551	0.815	1.523	2.523	10.937	0.680	1.523
fs09000000GET /search/authorities keyword = randomword	1300	0.361	0.016	0.551	0.827	1.679	2.680	12.588	0.707	1.679
fs09000000GET /search/authorities keyword == random sentence	1297	0.361	0.094	0.565	0.869	1.626	2.534	8.289	0.711	1.626
fs09000000GET /search/authorities keyword == randomword	1298	0.361	0.021	0.556	0.839	1.608	2.582	8.349	0.701	1.608
fs09000000GET /search/authorities keyword all randomword	1298	0.361	0.564	1.520	1.997	3.083	4.290	10.092	1.691	3.083
fs09000000GET /search/authorities keyword all random sentence	1297	0.361	0.105	0.549	0.829	1.657	2.655	8.973	0.688	1.657
fs09000000GET /search/authorities keyword all randomword	1300	0.361	0.017	0.564	0.830	1.617	2.824	9.527	0.714	1.617
fs09000000GET /search/authorities keyword any random sentence	1297	0.361	0.147	0.604	0.868	1.763	2.818	9.430	0.751	1.763
fs09000000GET /search/authorities keyword any randomword	1300	0.361	0.018	0.567	0.848	1.645	3.012	8.022	0.721	1.645
fs09000000GET /search/instances: contributors <> 2random words	1292	0.361	0.246	0.935	1.273	2.185	3.219	7.719	1.072	2.184
fs09000000GET /search/instances: contributors = *randomword	1293	0.360	0.672	1.611	2.000	2.919	4.005	13.528	1.764	2.919
fs09000000GET /search/instances: contributors = randomword*	1293	0.360	0.094	0.615	0.948	1.813	3.178	8.855	0.791	1.813
fs09000000GET /search/instances: contributors == 2random words	1292	0.361	0.118	0.542	0.809	1.567	2.535	9.101	0.675	1.567
fs09000000GET /search/instances: contributors all 2random words	1292	0.361	0.092	0.566	0.813	1.600	2.309	13.326	0.685	1.600
fs09000000GET /search/instances: contributors all randomword	1293	0.360	0.139	0.665	0.984	1.889	2.997	11.820	0.829	1.887
fs09000000GET /search/instances: contributors any 2random words	1292	0.361	0.152	0.692	0.977	1.842	3.031	8.169	0.844	1.841
fs09000000GET /search/instances: keyword <> 1word	1286	0.359	0.280	0.969	1.312	2.150	3.066	5.673	1.100	2.149
fs09000000GET /search/instances: keyword <> random 2words	1287	0.360	0.303	0.961	1.303	2.139	3.006	12.240	1.106	2.138
fs09000000GET /search/instances: keyword = 1word*	1287	0.360	0.168	0.738	1.034	1.908	2.872	9.941	0.885	1.907
fs09000000GET /search/instances: keyword = random 2words	1289	0.360	0.115	0.572	0.884	1.639	2.549	5.710	0.714	1.639
fs09000000GET /search/instances: keyword == random 2words	1287	0.360	0.131	0.570	0.870	1.759	2.572	8.373	0.725	1.759
fs09000000GET /search/instances: keyword all random 2words	1290	0.360	0.117	0.606	0.916	1.732	2.678	11.065	0.772	1.731
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang	1292	0.361	0.108	0.592	0.875	1.633	2.548	8.574	0.733	1.633
fs09000000GET /search/instances: keyword any random 2words	1289	0.360	0.269	0.775	1.113	1.960	2.838	8.873	0.931	1.959
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")	1292	0.361	0.188	0.741	1.035	1.821	2.858	11.866	0.868	1.819
fs09000000GET /search/instances: languages == lang	1292	0.361	0.175	0.686	0.983	1.827	2.765	5.447	0.823	1.826
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"	1292	0.361	0.164	0.778	1.089	1.927	3.108	12.475	0.922	1.926
fs09000000GET /search/instances: subjects <> 1randomword	1285	0.360	0.148	0.697	0.984	1.728	2.618	9.452	0.839	1.727
fs09000000GET /search/instances: subjects = (randomword OR randomword)	1294	0.360	0.200	0.735	1.032	1.852	3.152	11.812	0.892	1.847
fs09000000GET /search/instances: subjects = *1randomword	1285	0.360	0.923	2.121	2.557	3.713	5.281	8.462	2.252	3.712
fs09000000GET /search/instances: subjects = 1randomword*	1285	0.360	0.130	0.641	0.945	1.668	2.945	9.114	0.787	1.667
fs09000000GET /search/instances: subjects == 1randomword	1285	0.360	0.107	0.612	0.918	1.725	2.919	11.933	0.775	1.724
fs09000000GET /search/instances: subjects all 1randomword	1286	0.359	0.152	0.668	0.984	1.815	2.787	12.340	0.815	1.813
fs09000000GET /search/instances: subjects all randomword AND title all randomword	1294	0.360	0.118	0.605	0.873	1.727	2.714	6.724	0.739	1.727
fs09000000GET /search/instances: subjects all randomword NOT title all randomword	1294	0.360	0.123	0.738	1.064	1.852	2.876	5.605	0.882	1.851
fs09000000GET /search/instances: subjects all randomword OR title all randomword	1294	0.360	0.206	0.747	1.051	1.986	3.539	11.722	0.924	1.985
fs09000000GET /search/instances: subjects any 2random words	1285	0.360	0.131	0.691	0.967	1.839	2.728	9.587	0.839	1.837
fs09000000GET /search/instances: title <> 3 word sentence	1285	0.360	0.266	0.959	1.313	2.103	3.439	16.493	1.132	2.102
fs09000000GET /search/instances: title == 3 word sentence	1285	0.360	0.078	0.593	0.890	1.796	4.453	16.146	0.820	1.796
fs09000000GET /search/instances: title all *1randomword	1284	0.359	5.481	12.403	15.789	22.022	25.269	33.003	13.309	22.022
fs09000000GET /search/instances: title all 1randomword*	1284	0.360	0.120	0.682	0.981	1.988	3.632	12.476	0.865	1.987
fs09000000GET /search/instances: title all 3 word sentence	1285	0.360	0.097	0.590	0.864	1.766	3.475	12.152	0.779	1.766
fs09000000GET /search/instances: title any 3 word sentence	1285	0.360	0.326	1.109	1.942	4.373	6.041	12.549	1.612	4.371

Wildcard queries	Avg (seconds)	Error
fs09000000GET /search/instances: contributors = randomword*	0.791	0.0%
fs09000000GET /search/instances: keyword = 1word*	0.885	0.0%
fs09000000GET /search/instances: subjects = 1randomword*	0.787	0.0%
fs09000000GET /search/instances: title all 1randomword*	0.865	0.0%
	0.832

Wildcard prefix queries	Avg (seconds)	Error
fs09000000GET /search/authorities keyword = random sentence	2.045	0.0%
fs09000000GET /search/authorities keyword all randomword	1.691	0.0%
fs09000000GET /search/instances: contributors = *randomword	1.764	0.0%
fs09000000GET /search/instances: subjects = *1randomword	2.252	0.0%
fs09000000GET /search/instances: title all *1randomword	13.309	0.01% (2 failed requests)
	4.212

Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~30 seconds For all other queries it is closer to 10 seconds or less.

Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

Response time distribution

Out of ~69K requests, 32K requests had a response time > 700 milliseconds All these 32K requests were wildcard queries

CPU and Memory Utilization

CPU spikes linearly 40% as we start background reindexing. When we start the JMeter script run, the CPU continues to move upwards and reaches a max of 70%. Okapi also consumes resources but it is less than 40% which reasonable.

Memory is stable at around 65% throughout the test run and also during background reindexing

Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1650032755868&to=1650036953158

Longevity 24 hours test

Looking at CPU, Memory and Graphana dashboard, the longevity test for mod-search looks promising. mod-search is stable. Http request for wildcard prefix failed which is expected fs09000000GET /search/instances: title all *1randomword_GET_500

Also, not related to mod-search, there was an issue with the carrier io environment but other than the test looks good.

CPU and Memory Utilization

mod-search is around 40% which is relatively stable. CPU spiked to 40% from 30% after starting background ES reindexing.

Memory also looks stable at around 61%

JIRA tickets created as part of this investigation

MSEARCH-331 - Getting issue details... STATUS
MSEARCH-337 - Getting issue details... STATUS

Additional information

Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000

Artifact stored http://carrier-io.int.folio.ebsco.com/artifacts?q=mod-search - mod-search.zip

Kafka topics related to mod-search

{EnvID}.{tenant}.inventory.instance
{EnvID}.{tenant}.inventory.item
{EnvID}.{tenant}.inventory.holding-record
{EnvID}.{tenant}.inventory.bound-with (in new versions of mod-search\mod-inventory)

https://confluence.epnet.com/pages/viewpage.action?spaceKey=GOPAC&title=mod-search+performance+under+the+background+load

Browser not supported