PTF - mod search (Lotus)
Overview
- In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - - PERF-225Getting issue details... STATUS
We tested with 1, 20 virtual users against 10 million Inventory records.
Test plan:
- Run reindex for tenant A.
- Run test with n users for tenant B
Backend:
- mod-search-1.6.4
- mod-inventory-18.1.4
- mod-inventory-storage-23.0.2
- okapi-4.13.0
Frontend:
- folio_inventory-9.0.1
Environment:
- 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
- 80 FOLIO back-end modules deployed in 166 ECS services
- 3 okapi ECS services
- 7 m5.xlarge EC2 instances
- 1 writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance
High-Level Summary
Ran mod-search load testing for the following variant of queries:
- authority key search
- boolean search
- contributors search
- filter search
- keyword search
- subjects search
- title search
- wildcard search
- wildcard prefix search
mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.
Query | 1 user - Baseline (avg) | 1 user - Under the reindexing load (avg) | 20 user - Baseline (avg) | 20 user - Under the reindexing load (avg) |
Regular | 336 ms | 448 ms | 728 ms | 0.850 ms |
Wildcard | 1.42 seconds | 1.63 seconds | 2.32 seconds | 2.71 seconds (2 failed requests for /search/instances: title all *1randomword) |
*Note that "Regular" queries are queries that don't have the wild card character ( * ).
Show total requests processed increases as we increase the number of users concurrently.
1 user - Baseline (avg) ms | 1 user - Under the load (avg) ms | 20 user - Under the load (avg) ms | 20 user - Under the load (avg) ms | |
Total requests processed in 60 minutes test run | 10715 | 8051 | 69244 | 59420 |
Comparison with Kiwi
1 user | 1 user reindexing | 20 users | 20 users reindexing | |||||
Kiwi(ms) | Lotus(ms) | Kiwi(ms) | Lotus(ms) | Kiwi(ms) | Lotus(ms) | Kiwi(ms) | Lotus(ms) | |
Regular | 648 | 336 | 569 | 448 | 3190 | 728 | 7009 | 850 |
Wildcard queries | 660 | 268 | 561 | 159 | 4185 | 698 | 11160 | 832 |
Wildcard prefix queries | 13441 | 1740 | 11569 | 2657 | 17424 | 3624 | 22009 | 4212 |
1 user | 1 user reindexing | 20 users | 20 users reindexing | |||||
Percentage | Times better than Kiwi | Percentage | Times better than Kiwi | Percentage | Times better than Kiwi | Percentage | Times better than Kiwi | |
Regular Delta | 48.15% | 1 | 21.27% | 1 | 77.18% | 4 | 87.87% | 8 |
Wildcard queries Delta | 59.39% | 2 | 71.66% | 3 | 83.32% | 5 | 92.54% | 13 |
Wildcard prefix queries Delta | 87.05% | 7 | 77.03% | 4 | 79.20% | 4 | 80.86% | 5 |
CPU | Memory | Number of Tasks | ||||
Kiwi | Lotus | Kiwi | Lotus | Kiwi | Lotus | |
mod-search | 256 | 256 | 536 | 1440 | 2 | 1 |
okapi | 256 | 128 | 536 | 544 | 3 | 3 |
mod-authtoken | 128 | 128 | 360 | 322 | 2 | 2 |
mod-permissions | 128 | 128 | 536 | 537 | 2 | 2 |
Elastic Search | No differences |
Test Runs
1 User without any background reindexing
- runtime 60 minutes
Requests | Total | Req/s | Min | 50pct | 75pct | 95pct | 99pct | Max | Avg | Latency |
fs09000000GET /search/authorities keyword <> random sentence | 233 | 0.065 | 0.250 | 0.264 | 0.298 | 0.520 | 0.721 | 0.973 | 0.302 | 0.520 |
fs09000000GET /search/authorities keyword <> randomword | 233 | 0.065 | 0.246 | 0.261 | 0.289 | 0.548 | 0.810 | 1.771 | 0.312 | 0.548 |
fs09000000GET /search/authorities keyword = *random sentence* | 233 | 0.065 | 0.567 | 0.710 | 0.962 | 1.681 | 2.240 | 2.582 | 0.874 | 1.681 |
fs09000000GET /search/authorities keyword = random sentence | 233 | 0.065 | 0.017 | 0.021 | 0.023 | 0.034 | 0.048 | 0.074 | 0.023 | 0.034 |
fs09000000GET /search/authorities keyword = randomword | 233 | 0.065 | 0.014 | 0.024 | 0.056 | 0.135 | 0.204 | 0.449 | 0.044 | 0.135 |
fs09000000GET /search/authorities keyword == random sentence | 233 | 0.065 | 0.017 | 0.020 | 0.022 | 0.035 | 0.055 | 0.101 | 0.022 | 0.033 |
fs09000000GET /search/authorities keyword == randomword | 233 | 0.065 | 0.015 | 0.022 | 0.034 | 0.089 | 0.164 | 0.241 | 0.034 | 0.089 |
fs09000000GET /search/authorities keyword all *randomword* | 233 | 0.065 | 0.302 | 0.554 | 0.824 | 1.198 | 1.675 | 2.013 | 0.644 | 1.198 |
fs09000000GET /search/authorities keyword all random sentence | 233 | 0.065 | 0.017 | 0.021 | 0.025 | 0.040 | 0.076 | 0.230 | 0.025 | 0.040 |
fs09000000GET /search/authorities keyword all randomword | 233 | 0.065 | 0.025 | 0.047 | 0.089 | 0.158 | 0.270 | 0.379 | 0.067 | 0.158 |
fs09000000GET /search/authorities keyword any random sentence | 233 | 0.065 | 0.017 | 0.092 | 0.129 | 0.200 | 0.315 | 0.354 | 0.098 | 0.199 |
fs09000000GET /search/authorities keyword any randomword | 233 | 0.065 | 0.014 | 0.028 | 0.062 | 0.145 | 0.197 | 0.228 | 0.046 | 0.145 |
fs09000000GET /search/instances: contributors <> 2random words | 233 | 0.065 | 0.255 | 0.274 | 0.296 | 0.422 | 0.752 | 1.599 | 0.302 | 0.420 |
fs09000000GET /search/instances: contributors = *randomword | 233 | 0.065 | 0.449 | 0.481 | 0.570 | 0.943 | 1.357 | 1.790 | 0.562 | 0.942 |
fs09000000GET /search/instances: contributors = randomword* | 233 | 0.065 | 0.013 | 0.041 | 0.160 | 0.219 | 0.242 | 0.276 | 0.084 | 0.218 |
fs09000000GET /search/instances: contributors == 2random words | 233 | 0.065 | 0.013 | 0.020 | 0.026 | 0.040 | 0.069 | 0.078 | 0.023 | 0.039 |
fs09000000GET /search/instances: contributors all 2random words | 233 | 0.065 | 0.014 | 0.017 | 0.024 | 0.038 | 0.136 | 0.211 | 0.023 | 0.038 |
fs09000000GET /search/instances: contributors all randomword | 233 | 0.065 | 0.014 | 0.089 | 0.179 | 0.240 | 0.364 | 0.441 | 0.112 | 0.238 |
fs09000000GET /search/instances: contributors any 2random words | 233 | 0.065 | 0.017 | 0.152 | 0.196 | 0.240 | 0.390 | 0.622 | 0.139 | 0.239 |
fs09000000GET /search/instances: keyword <> 1word | 233 | 0.065 | 0.258 | 0.282 | 0.324 | 0.489 | 0.709 | 1.219 | 0.323 | 0.488 |
fs09000000GET /search/instances: keyword <> random 2words | 233 | 0.065 | 0.259 | 0.281 | 0.330 | 0.570 | 0.725 | 0.803 | 0.326 | 0.569 |
fs09000000GET /search/instances: keyword = 1word* | 233 | 0.065 | 0.018 | 0.170 | 0.207 | 0.274 | 0.456 | 0.591 | 0.159 | 0.273 |
fs09000000GET /search/instances: keyword = random 2words | 233 | 0.065 | 0.016 | 0.051 | 0.082 | 0.153 | 0.299 | 0.313 | 0.063 | 0.153 |
fs09000000GET /search/instances: keyword == random 2words | 233 | 0.065 | 0.016 | 0.028 | 0.041 | 0.082 | 0.122 | 0.370 | 0.037 | 0.082 |
fs09000000GET /search/instances: keyword all random 2words | 233 | 0.065 | 0.018 | 0.059 | 0.096 | 0.198 | 0.397 | 0.434 | 0.080 | 0.197 |
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang | 233 | 0.065 | 0.020 | 0.075 | 0.121 | 0.194 | 0.289 | 0.462 | 0.093 | 0.194 |
fs09000000GET /search/instances: keyword any random 2words | 233 | 0.065 | 0.049 | 0.203 | 0.237 | 0.343 | 0.599 | 1.201 | 0.220 | 0.342 |
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit") | 233 | 0.065 | 0.043 | 0.057 | 0.172 | 0.237 | 0.377 | 0.453 | 0.105 | 0.236 |
fs09000000GET /search/instances: languages == lang | 233 | 0.065 | 0.043 | 0.054 | 0.148 | 0.185 | 0.358 | 0.461 | 0.094 | 0.184 |
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available" | 233 | 0.065 | 0.047 | 0.070 | 0.213 | 0.244 | 0.494 | 0.618 | 0.127 | 0.243 |
fs09000000GET /search/instances: subjects <> 1randomword | 233 | 0.065 | 0.015 | 0.183 | 0.219 | 0.259 | 0.357 | 0.697 | 0.169 | 0.258 |
fs09000000GET /search/instances: subjects = (randomword OR randomword) | 233 | 0.065 | 0.023 | 0.173 | 0.216 | 0.258 | 0.407 | 0.453 | 0.167 | 0.257 |
fs09000000GET /search/instances: subjects = *1randomword | 233 | 0.065 | 0.647 | 0.743 | 0.872 | 1.327 | 2.303 | 2.916 | 0.850 | 1.325 |
fs09000000GET /search/instances: subjects = 1randomword* | 233 | 0.065 | 0.013 | 0.116 | 0.158 | 0.210 | 0.270 | 0.289 | 0.104 | 0.209 |
fs09000000GET /search/instances: subjects == 1randomword | 233 | 0.065 | 0.014 | 0.054 | 0.109 | 0.201 | 0.245 | 0.390 | 0.081 | 0.199 |
fs09000000GET /search/instances: subjects all 1randomword | 233 | 0.065 | 0.015 | 0.072 | 0.160 | 0.229 | 0.304 | 0.353 | 0.099 | 0.227 |
fs09000000GET /search/instances: subjects all randomword AND title all randomword | 233 | 0.065 | 0.015 | 0.049 | 0.073 | 0.130 | 0.245 | 0.375 | 0.059 | 0.130 |
fs09000000GET /search/instances: subjects all randomword NOT title all randomword | 233 | 0.065 | 0.015 | 0.187 | 0.236 | 0.334 | 0.514 | 0.745 | 0.182 | 0.333 |
fs09000000GET /search/instances: subjects all randomword OR title all randomword | 233 | 0.065 | 0.026 | 0.213 | 0.256 | 0.325 | 0.461 | 0.528 | 0.214 | 0.324 |
fs09000000GET /search/instances: subjects any 2random words | 233 | 0.065 | 0.016 | 0.132 | 0.176 | 0.245 | 0.459 | 0.802 | 0.141 | 0.243 |
fs09000000GET /search/instances: title <> 3 word sentence | 232 | 0.065 | 0.260 | 0.282 | 0.314 | 0.499 | 0.692 | 0.751 | 0.320 | 0.498 |
fs09000000GET /search/instances: title == 3 word sentence | 232 | 0.065 | 0.017 | 0.031 | 0.050 | 0.110 | 0.194 | 0.230 | 0.043 | 0.110 |
fs09000000GET /search/instances: title all *1randomword | 232 | 0.065 | 4.602 | 5.811 | 8.438 | 11.234 | 14.750 | 15.833 | 6.886 | 11.234 |
fs09000000GET /search/instances: title all 1randomword* | 232 | 0.065 | 0.017 | 0.085 | 0.174 | 0.236 | 0.380 | 0.622 | 0.117 | 0.234 |
fs09000000GET /search/instances: title all 3 word sentence | 233 | 0.065 | 0.020 | 0.066 | 0.115 | 0.188 | 0.264 | 0.298 | 0.084 | 0.188 |
fs09000000GET /search/instances: title any 3 word sentence | 233 | 0.065 | 0.116 | 0.313 | 0.690 | 1.747 | 2.429 | 2.833 | 0.585 | 1.745 |
Wildcard queries | Avg (seconds) | Error |
fs09000000GET /search/instances: contributors = randomword* | 0.084 | 0.0% |
fs09000000GET /search/instances: keyword = 1word* | 0.159 | 0.0% |
fs09000000GET /search/instances: subjects = 1randomword* | 0.104 | 0.0% |
fs09000000GET /search/instances: title all 1randomword* | 0.117 | 0.0% |
0.2676 |
Wildcard prefix queries | Avg (seconds) | Error |
fs09000000GET /search/authorities keyword = *random sentence* | 0.874 | 0.0% |
fs09000000GET /search/authorities keyword all *randomword* | 0.644 | 0.0% |
fs09000000GET /search/instances: contributors = *randomword | 0.562 | 0.0% |
fs09000000GET /search/authorities keyword all *randomword* | 0.644 | 0.0% |
fs09000000GET /search/instances: subjects = *1randomword | 0.85 | 0.0% |
fs09000000GET /search/instances: title all *1randomword | 6.886 | 0.0% |
1.74 |
Response time over time
Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.
Throughput
Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.
Response time distribution
Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries
- Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649439816854&to=1649444031072
CPU and Memory Utilization
mod-search's CPU was reasonable at around 10% throughout the test run
mod-search memory is stable around 40% throughtout the test run
1 User with background reindexing
- runtime 60 minutes
Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18%
Requests | Total | Req/s | Min | 50pct | 75pct | 95pct | 99pct | Max | Avg | Latency |
fs09000000GET /search/authorities keyword <> random sentence | 175 | 0.049 | 0.088 | 0.221 | 0.339 | 0.557 | 0.685 | 1.104 | 0.263 | 0.557 |
fs09000000GET /search/authorities keyword <> randomword | 175 | 0.049 | 0.087 | 0.158 | 0.289 | 0.505 | 0.592 | 1.069 | 0.216 | 0.505 |
fs09000000GET /search/authorities keyword = *random sentence* | 175 | 0.049 | 0.570 | 0.917 | 1.203 | 1.708 | 2.337 | 3.768 | 1.013 | 1.708 |
fs09000000GET /search/authorities keyword = random sentence | 175 | 0.049 | 0.017 | 0.021 | 0.028 | 0.046 | 0.067 | 0.106 | 0.026 | 0.046 |
fs09000000GET /search/authorities keyword = randomword | 175 | 0.049 | 0.015 | 0.028 | 0.054 | 0.140 | 0.186 | 0.342 | 0.046 | 0.140 |
fs09000000GET /search/authorities keyword == random sentence | 175 | 0.049 | 0.017 | 0.021 | 0.029 | 0.068 | 0.167 | 0.210 | 0.029 | 0.068 |
fs09000000GET /search/authorities keyword == randomword | 175 | 0.049 | 0.015 | 0.026 | 0.040 | 0.150 | 0.229 | 0.271 | 0.042 | 0.149 |
fs09000000GET /search/authorities keyword all *randomword* | 175 | 0.049 | 0.311 | 0.657 | 0.839 | 1.269 | 1.490 | 3.562 | 0.720 | 1.269 |
fs09000000GET /search/authorities keyword all random sentence | 175 | 0.049 | 0.017 | 0.026 | 0.035 | 0.073 | 0.137 | 0.198 | 0.033 | 0.073 |
fs09000000GET /search/authorities keyword all randomword | 175 | 0.049 | 0.025 | 0.047 | 0.120 | 0.195 | 0.279 | 0.652 | 0.083 | 0.195 |
fs09000000GET /search/authorities keyword any random sentence | 175 | 0.049 | 0.018 | 0.145 | 0.190 | 0.342 | 0.408 | 0.425 | 0.149 | 0.342 |
fs09000000GET /search/authorities keyword any randomword | 175 | 0.049 | 0.014 | 0.030 | 0.090 | 0.208 | 0.244 | 0.387 | 0.062 | 0.208 |
fs09000000GET /search/instances: contributors <> 2random words | 175 | 0.049 | 0.144 | 0.313 | 0.387 | 0.668 | 0.773 | 0.969 | 0.339 | 0.667 |
fs09000000GET /search/instances: contributors = *randomword | 175 | 0.049 | 0.460 | 0.739 | 0.940 | 1.445 | 2.064 | 2.245 | 0.830 | 1.444 |
fs09000000GET /search/instances: contributors = randomword* | 175 | 0.049 | 0.014 | 0.077 | 0.213 | 0.313 | 0.374 | 0.407 | 0.119 | 0.312 |
fs09000000GET /search/instances: contributors == 2random words | 175 | 0.049 | 0.014 | 0.027 | 0.036 | 0.061 | 0.109 | 0.208 | 0.032 | 0.061 |
fs09000000GET /search/instances: contributors all 2random words | 175 | 0.049 | 0.014 | 0.020 | 0.028 | 0.048 | 0.058 | 0.072 | 0.025 | 0.048 |
fs09000000GET /search/instances: contributors all randomword | 175 | 0.049 | 0.015 | 0.143 | 0.257 | 0.325 | 0.421 | 0.607 | 0.159 | 0.325 |
fs09000000GET /search/instances: contributors any 2random words | 175 | 0.049 | 0.019 | 0.205 | 0.272 | 0.381 | 0.437 | 0.593 | 0.198 | 0.379 |
fs09000000GET /search/instances: keyword <> 1word | 175 | 0.049 | 0.144 | 0.324 | 0.429 | 0.653 | 0.769 | 1.073 | 0.352 | 0.652 |
fs09000000GET /search/instances: keyword <> random 2words | 175 | 0.049 | 0.149 | 0.316 | 0.418 | 0.690 | 1.037 | 1.177 | 0.365 | 0.688 |
fs09000000GET /search/instances: keyword = 1word* | 175 | 0.049 | 0.018 | 0.228 | 0.267 | 0.377 | 0.587 | 0.799 | 0.217 | 0.376 |
fs09000000GET /search/instances: keyword = random 2words | 175 | 0.049 | 0.017 | 0.072 | 0.123 | 0.198 | 0.271 | 0.421 | 0.087 | 0.198 |
fs09000000GET /search/instances: keyword == random 2words | 175 | 0.049 | 0.017 | 0.032 | 0.074 | 0.178 | 0.303 | 0.345 | 0.060 | 0.178 |
fs09000000GET /search/instances: keyword all random 2words | 175 | 0.049 | 0.019 | 0.106 | 0.156 | 0.271 | 0.360 | 0.435 | 0.125 | 0.271 |
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang | 175 | 0.049 | 0.048 | 0.169 | 0.226 | 0.306 | 0.492 | 0.807 | 0.181 | 0.306 |
fs09000000GET /search/instances: keyword any random 2words | 175 | 0.049 | 0.054 | 0.282 | 0.372 | 0.560 | 0.796 | 1.031 | 0.324 | 0.559 |
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit") | 175 | 0.049 | 0.045 | 0.081 | 0.183 | 0.284 | 0.408 | 0.441 | 0.125 | 0.283 |
fs09000000GET /search/instances: languages == lang | 175 | 0.049 | 0.044 | 0.081 | 0.164 | 0.299 | 0.379 | 0.519 | 0.120 | 0.298 |
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available" | 175 | 0.049 | 0.049 | 0.108 | 0.234 | 0.423 | 0.615 | 0.676 | 0.167 | 0.422 |
fs09000000GET /search/instances: subjects <> 1randomword | 175 | 0.049 | 0.016 | 0.249 | 0.281 | 0.380 | 0.448 | 0.555 | 0.228 | 0.379 |
fs09000000GET /search/instances: subjects = (randomword OR randomword) | 175 | 0.049 | 0.024 | 0.248 | 0.287 | 0.363 | 0.464 | 0.682 | 0.242 | 0.362 |
fs09000000GET /search/instances: subjects = *1randomword | 175 | 0.049 | 0.693 | 1.071 | 1.305 | 1.849 | 2.368 | 2.855 | 1.156 | 1.848 |
fs09000000GET /search/instances: subjects = 1randomword* | 175 | 0.049 | 0.014 | 0.169 | 0.218 | 0.283 | 0.308 | 0.576 | 0.144 | 0.281 |
fs09000000GET /search/instances: subjects == 1randomword | 175 | 0.049 | 0.015 | 0.076 | 0.159 | 0.288 | 0.366 | 0.565 | 0.111 | 0.287 |
fs09000000GET /search/instances: subjects all 1randomword | 175 | 0.049 | 0.014 | 0.112 | 0.212 | 0.297 | 0.397 | 1.243 | 0.142 | 0.296 |
fs09000000GET /search/instances: subjects all randomword AND title all randomword | 175 | 0.049 | 0.023 | 0.088 | 0.122 | 0.200 | 0.297 | 0.511 | 0.100 | 0.200 |
fs09000000GET /search/instances: subjects all randomword NOT title all randomword | 175 | 0.049 | 0.023 | 0.279 | 0.341 | 0.445 | 0.557 | 0.709 | 0.265 | 0.444 |
fs09000000GET /search/instances: subjects all randomword OR title all randomword | 175 | 0.049 | 0.068 | 0.310 | 0.358 | 0.521 | 0.728 | 0.836 | 0.326 | 0.520 |
fs09000000GET /search/instances: subjects any 2random words | 175 | 0.049 | 0.020 | 0.206 | 0.262 | 0.345 | 0.495 | 0.597 | 0.201 | 0.344 |
fs09000000GET /search/instances: title <> 3 word sentence | 175 | 0.049 | 0.152 | 0.347 | 0.458 | 0.771 | 1.131 | 1.180 | 0.397 | 0.770 |
fs09000000GET /search/instances: title == 3 word sentence | 175 | 0.049 | 0.017 | 0.037 | 0.120 | 0.204 | 0.359 | 0.413 | 0.079 | 0.204 |
fs09000000GET /search/instances: title all *1randomword | 175 | 0.049 | 5.006 | 8.103 | 11.273 | 18.093 | 22.114 | 22.748 | 9.568 | 18.093 |
fs09000000GET /search/instances: title all 1randomword* | 175 | 0.049 | 0.017 | 0.113 | 0.245 | 0.391 | 0.504 | 0.639 | 0.159 | 0.390 |
fs09000000GET /search/instances: title all 3 word sentence | 175 | 0.049 | 0.039 | 0.133 | 0.178 | 0.274 | 0.317 | 0.777 | 0.148 | 0.274 |
fs09000000GET /search/instances: title any 3 word sentence | 175 | 0.049 | 0.175 | 0.473 | 0.995 | 2.822 | 3.720 | 4.922 | 0.860 | 2.821 |
Wildcard queries | Avg (seconds) | Error |
fs09000000GET /search/instances: contributors = randomword* | 0.119 | 0.0% |
fs09000000GET /search/instances: keyword = 1word* | 0.217 | 0.0% |
fs09000000GET /search/instances: subjects = 1randomword* | 0.144 | 0.0% |
fs09000000GET /search/instances: title all 1randomword* | 0.159 | 0.0% |
0.159 |
Wildcard prefix queries | Avg (seconds) | Error |
fs09000000GET /search/authorities keyword = *random sentence* | 1.013 | 0.0% |
fs09000000GET /search/authorities keyword all *randomword* | 0.720 | 0.0% |
fs09000000GET /search/instances: contributors = *randomword | 0.830 | 0.0% |
fs09000000GET /search/instances: subjects = *1randomword | 1.156 | 0.0% |
fs09000000GET /search/instances: title all *1randomword | 9.568 | 0.0% |
2.657 |
Response Time Over Time
Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.
Throughput
Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing
Response Time distribution
Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing
CPU and Memory Utilization
When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.
mod-search memory is stable at around 65% throughout the test run and also during background reindexing
20 Users without any background reindexing
- runtime 60 minutes
Requests | Total | Req/s | Min | 50pct | 75pct | 95pct | 99pct | Max | Avg | Latency |
fs09000000GET /search/authorities keyword <> random sentence | 1508 | 0.420 | 0.149 | 0.723 | 1.042 | 1.781 | 2.589 | 6.343 | 0.850 | 1.781 |
fs09000000GET /search/authorities keyword <> randomword | 1511 | 0.420 | 0.156 | 0.713 | 1.030 | 1.840 | 2.887 | 5.796 | 0.851 | 1.836 |
fs09000000GET /search/authorities keyword = *random sentence* | 1508 | 0.420 | 0.883 | 1.671 | 2.048 | 2.847 | 4.687 | 8.326 | 1.788 | 2.847 |
fs09000000GET /search/authorities keyword = random sentence | 1508 | 0.420 | 0.031 | 0.460 | 0.717 | 1.376 | 2.219 | 10.218 | 0.596 | 1.376 |
fs09000000GET /search/authorities keyword = randomword | 1511 | 0.420 | 0.026 | 0.449 | 0.711 | 1.406 | 2.594 | 6.043 | 0.593 | 1.404 |
fs09000000GET /search/authorities keyword == random sentence | 1508 | 0.420 | 0.068 | 0.447 | 0.693 | 1.278 | 2.153 | 5.411 | 0.558 | 1.278 |
fs09000000GET /search/authorities keyword == randomword | 1510 | 0.420 | 0.016 | 0.472 | 0.734 | 1.561 | 2.577 | 7.378 | 0.614 | 1.561 |
fs09000000GET /search/authorities keyword all *randomword* | 1510 | 0.419 | 0.545 | 1.359 | 1.743 | 2.545 | 3.722 | 6.997 | 1.468 | 2.545 |
fs09000000GET /search/authorities keyword all random sentence | 1509 | 0.419 | 0.053 | 0.471 | 0.722 | 1.361 | 2.491 | 5.385 | 0.591 | 1.361 |
fs09000000GET /search/authorities keyword all randomword | 1512 | 0.420 | 0.021 | 0.486 | 0.771 | 1.603 | 2.648 | 5.197 | 0.632 | 1.603 |
fs09000000GET /search/authorities keyword any random sentence | 1508 | 0.420 | 0.051 | 0.507 | 0.767 | 1.464 | 2.614 | 5.463 | 0.636 | 1.464 |
fs09000000GET /search/authorities keyword any randomword | 1512 | 0.420 | 0.019 | 0.485 | 0.759 | 1.546 | 2.559 | 4.932 | 0.613 | 1.546 |
fs09000000GET /search/instances: contributors <> 2random words | 1506 | 0.420 | 0.229 | 0.845 | 1.202 | 2.095 | 3.866 | 10.696 | 0.998 | 2.094 |
fs09000000GET /search/instances: contributors = *randomword | 1506 | 0.420 | 0.602 | 1.360 | 1.728 | 2.526 | 3.411 | 4.899 | 1.472 | 2.526 |
fs09000000GET /search/instances: contributors = randomword* | 1506 | 0.420 | 0.064 | 0.508 | 0.778 | 1.548 | 2.319 | 6.598 | 0.642 | 1.547 |
fs09000000GET /search/instances: contributors == 2random words | 1506 | 0.420 | 0.065 | 0.459 | 0.702 | 1.401 | 2.373 | 6.619 | 0.588 | 1.401 |
fs09000000GET /search/instances: contributors all 2random words | 1506 | 0.420 | 0.058 | 0.472 | 0.749 | 1.424 | 2.618 | 5.975 | 0.607 | 1.424 |
fs09000000GET /search/instances: contributors all randomword | 1508 | 0.420 | 0.042 | 0.550 | 0.813 | 1.512 | 2.460 | 6.007 | 0.670 | 1.512 |
fs09000000GET /search/instances: contributors any 2random words | 1506 | 0.420 | 0.090 | 0.590 | 0.860 | 1.558 | 2.395 | 11.632 | 0.726 | 1.556 |
fs09000000GET /search/instances: keyword <> 1word | 1503 | 0.419 | 0.219 | 0.890 | 1.219 | 2.047 | 2.918 | 10.988 | 1.012 | 2.046 |
fs09000000GET /search/instances: keyword <> random 2words | 1504 | 0.419 | 0.215 | 0.852 | 1.190 | 1.924 | 2.871 | 6.320 | 0.975 | 1.922 |
fs09000000GET /search/instances: keyword = 1word* | 1503 | 0.419 | 0.065 | 0.598 | 0.867 | 1.558 | 2.587 | 4.883 | 0.728 | 1.557 |
fs09000000GET /search/instances: keyword = random 2words | 1504 | 0.419 | 0.077 | 0.472 | 0.714 | 1.436 | 2.553 | 4.621 | 0.601 | 1.434 |
fs09000000GET /search/instances: keyword == random 2words | 1504 | 0.419 | 0.079 | 0.489 | 0.771 | 1.417 | 2.254 | 6.359 | 0.614 | 1.417 |
fs09000000GET /search/instances: keyword all random 2words | 1505 | 0.419 | 0.057 | 0.526 | 0.770 | 1.453 | 2.730 | 9.947 | 0.649 | 1.453 |
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang | 1506 | 0.420 | 0.066 | 0.492 | 0.740 | 1.497 | 2.171 | 4.692 | 0.619 | 1.484 |
fs09000000GET /search/instances: keyword any random 2words | 1504 | 0.419 | 0.188 | 0.661 | 0.941 | 1.595 | 2.520 | 7.017 | 0.790 | 1.594 |
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit") | 1505 | 0.419 | 0.103 | 0.612 | 0.903 | 1.685 | 2.572 | 8.275 | 0.746 | 1.681 |
fs09000000GET /search/instances: languages == lang | 1505 | 0.419 | 0.106 | 0.584 | 0.876 | 1.530 | 2.430 | 7.004 | 0.712 | 1.528 |
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available" | 1506 | 0.419 | 0.099 | 0.638 | 0.929 | 1.727 | 2.791 | 7.607 | 0.778 | 1.720 |
fs09000000GET /search/instances: subjects <> 1randomword | 1498 | 0.418 | 0.082 | 0.589 | 0.867 | 1.665 | 3.147 | 14.882 | 0.744 |