MSEARCH-300 - Getting issue details... STATUS
Environment
Summary
Test 1 user call number browsing with random words.
Test 5 users call number browsing with random words.
Test 10 users call number browsing with random words.
Test 20 users call number browsing with random words.
Test 50 users call number browsing with random words.
- Test 50 users call number browsing with random words with Data Import in background.
- Test 50 users call number browsing with real keywords from database with Data Import in background.
- Comparison chart - call number browsing real keys + Data Import in background
- Test 50 users call number browsing + DI + CICO
Test with long ramp-up period, 1 minute per user from 0 to 25 users
Environment
- mod-search 1.6.4
- OpenSearch service
nodes type r6g.large.search (X4) * (4 data nodes 3 dedicated master nodes);
- version OpenSearch 1.2 (latest);
- kafka 2.8.0
Summary
- ALL tests (1-50 users) ended without errors. Average response time . most resource intensive is mod-search. During test 50 users call number browsing CPU usage reached more than 300%.
- During tests 50 users with background activities (CICO+ Data Import) response time increased up to 10%-15%.
| number of users | average rt | % CPU mod-search |
|---|---|---|
| 1 | 0.148 | 12 |
| 5 | 0.168 | 42 |
| 10 | 0.191 | 75 |
| 20 | 0.321 | 122 |
| 50 | 1.821 | 158 |
1 user
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 1747 | 1747 | 0 | 0 | 2.220 | 0.053 | 0.142 | 0.166 | 0.255 | 0.308 | 0.687 | 0.148 | 0.251 |
5 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 8219 | 8219 | 0 | 0 | 10.693 | 0.048 | 0.150 | 0.188 | 0.295 | 0.419 | 0.920 | 0.168 | 0.292 |
10 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 12741 | 12741 | 0 | 0 | 20.313 | 0.051 | 0.161 | 0.233 | 0.387 | 0.584 | 1.437 | 0.191 | 0.384 |
20 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 12193 | 12193 | 0 | 0 | 32.169 | 0.052 | 0.245 | 0.441 | 0.739 | 1.025 | 2.441 | 0.321 | 0.736 |
50 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 7849 | 7849 | 0 | 0 | 38.485 | 0.081 | 0.963 | 1.259 | 1.823 | 2.182 | 3.275 | 0.996 | 1.821 |
50 users 30 min (10 min ramp up)
50 users + DI in background
50 users +DI 5K in background
RTT without background activity
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers "BT83.57 .N625 2013" | 10276 | 10276 | 0 | 0 | 9.349 | 0.046 | 0.161 | 0.234 | 0.401 | 0.779 | 1.547 | 0.192 | 0.401 |
| GET_browse _call_numbers "CC76.4 .C66 2013" | 10275 | 10275 | 0 | 0 | 9.352 | 0.046 | 0.169 | 0.243 | 0.402 | 0.806 | 1.736 | 0.199 | 0.402 |
| GET_browse _call_numbers "DT107.827.L32 1987" | 10277 | 10277 | 0 | 0 | 9.346 | 0.050 | 0.169 | 0.242 | 0.400 | 0.810 | 1.721 | 0.198 | 0.399 |
| GET_browse _call_numbers "G3893.M5 1923 .W4" | 10275 | 10275 | 0 | 0 | 9.336 | 0.053 | 0.173 | 0.248 | 0.407 | 0.821 | 1.753 | 0.203 | 0.407 |
| GET_browse _call_numbers "LC4019 .H83 2012" | 10275 | 10275 | 0 | 0 | 9.355 | 0.056 | 0.181 | 0.253 | 0.418 | 0.823 | 1.606 | 0.210 | 0.418 |
| GET_browse _call_numbers "N6370 .C585 2013eb" | 10276 | 10276 | 0 | 0 | 9.343 | 0.054 | 0.170 | 0.243 | 0.397 | 0.763 | 1.847 | 0.199 | 0.397 |
| GET_browse _call_numbers "P302.77 .P67 2013" | 10273 | 10273 | 0 | 0 | 9.349 | 0.062 | 0.192 | 0.267 | 0.433 | 0.808 | 1.880 | 0.222 | 0.433 |
| GET_browse _call_numbers "PR2274 .P7 1633" | 10277 | 10277 | 0 | 0 | 9.334 | 0.047 | 0.168 | 0.245 | 0.411 | 0.811 | 1.815 | 0.201 | 0.411 |
| GET_browse _call_numbers "TL220 .B85 2012" | 10276 | 10276 | 0 | 0 | 9.339 | 0.043 | 0.151 | 0.224 | 0.383 | 0.734 | 1.765 | 0.182 | 0.383 |
| GET_browse _call_numbers "Z711 .C88 2013" | 10275 | 10275 | 0 | 0 | 9.350 | 0.093 | 0.260 | 0.339 | 0.507 | 0.995 | 2.074 | 0.289 | 0.507 |
RTT with DI in the background.
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers "BT83.57 .N625 2013" | 2072 | 2072 | 0 | 0 | 9.280 | 0.054 | 0.211 | 0.293 | 0.460 | 0.755 | 1.423 | 0.239 | 0.460 |
| GET_browse _call_numbers "CC76.4 .C66 2013" | 2073 | 2073 | 0 | 0 | 9.282 | 0.051 | 0.224 | 0.306 | 0.469 | 0.717 | 1.419 | 0.249 | 0.469 |
| GET_browse _call_numbers "DT107.827.L32 1987" | 2073 | 2073 | 0 | 0 | 9.277 | 0.053 | 0.216 | 0.297 | 0.461 | 0.684 | 1.359 | 0.241 | 0.460 |
| GET_browse _call_numbers "G3893.M5 1923 .W4" | 2073 | 2073 | 0 | 0 | 9.270 | 0.058 | 0.217 | 0.302 | 0.462 | 0.725 | 1.332 | 0.244 | 0.462 |
| GET_browse _call_numbers "LC4019 .H83 2012" | 2071 | 2071 | 0 | 0 | 9.283 | 0.067 | 0.234 | 0.314 | 0.484 | 0.742 | 1.330 | 0.261 | 0.484 |
| GET_browse _call_numbers "N6370 .C585 2013eb" | 2073 | 2073 | 0 | 0 | 9.275 | 0.060 | 0.218 | 0.298 | 0.464 | 0.690 | 1.454 | 0.246 | 0.464 |
| GET_browse _call_numbers "P302.77 .P67 2013" | 2071 | 2071 | 0 | 0 | 9.278 | 0.068 | 0.240 | 0.325 | 0.500 | 0.752 | 1.484 | 0.268 | 0.500 |
| GET_browse _call_numbers "PR2274 .P7 1633" | 2074 | 2074 | 0 | 0 | 9.266 | 0.049 | 0.221 | 0.304 | 0.464 | 0.691 | 1.485 | 0.245 | 0.464 |
| GET_browse _call_numbers "TL220 .B85 2012" | 2072 | 2072 | 0 | 0 | 9.273 | 0.044 | 0.198 | 0.274 | 0.438 | 0.671 | 1.449 | 0.225 | 0.437 |
| GET_browse _call_numbers "Z711 .C88 2013" | 2075 | 2075 | 0 | 0 | 9.282 | 0.103 | 0.315 | 0.406 | 0.591 | 0.871 | 1.604 | 0.340 | 0.591 |
| average real call numbers from database | 0.2095 |
| average random words | 0.326 |
Comparison chart - call number browsing real keys + data import 5K in background
Call number browsing with background CICO activity.
A large usege of processor resources is observed in the mod-search module during call number browsing. I guess need to increase the processor resources in tack definition of this module.
Grafana chart from test - random keywords.
Grafana chart from test - real keywords from database.
SELECT jsonb -> effectiveCallNumberComponents →> 'callNumber') FROM fs09000000_mod_inventory_storage.item WHERE jsonb -> effectiveCallNumberComponents →'callNumber' is not null limit 10
50 users call number browsing random words + 10 user CICO in background
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers BT83.57 .N625 2013 | 5299 | 5299 | 0 | 0 | 8.566 | 0.048 | 0.172 | 0.268 | 0.475 | 0.802 | 2.981 | 0.214 | 0.475 |
| GET_browse _call_numbers CC76.4 .C66 2013 | 5297 | 5297 | 0 | 0 | 8.569 | 0.050 | 0.178 | 0.282 | 0.486 | 0.884 | 2.249 | 0.224 | 0.485 |
| GET_browse _call_numbers DT107.827.L32 1987 | 5297 | 5297 | 0 | 0 | 8.561 | 0.052 | 0.181 | 0.281 | 0.502 | 0.922 | 2.485 | 0.226 | 0.502 |
| GET_browse _call_numbers G3893.M5 1923 .W4 | 5297 | 5297 | 0 | 0 | 8.549 | 0.054 | 0.182 | 0.289 | 0.511 | 0.882 | 2.742 | 0.230 | 0.511 |
| GET_browse _call_numbers LC4019 .H83 2012 | 5298 | 5298 | 0 | 0 | 8.576 | 0.058 | 0.193 | 0.290 | 0.513 | 0.879 | 2.337 | 0.237 | 0.513 |
| GET_browse _call_numbers N6370 .C585 2013eb | 5296 | 5296 | 0 | 0 | 8.556 | 0.054 | 0.178 | 0.272 | 0.491 | 0.869 | 2.302 | 0.223 | 0.491 |
| GET_browse _call_numbers P302.77 .P67 2013 | 5297 | 5297 | 0 | 0 | 8.562 | 0.066 | 0.204 | 0.310 | 0.527 | 0.862 | 2.916 | 0.250 | 0.527 |
| GET_browse _call_numbers PR2274 .P7 1633 | 5299 | 5299 | 0 | 0 | 8.547 | 0.048 | 0.182 | 0.280 | 0.504 | 0.903 | 2.142 | 0.228 | 0.504 |
| GET_browse _call_numbers TL220 .B85 2012 | 5297 | 5297 | 0 | 0 | 8.552 | 0.043 | 0.162 | 0.259 | 0.466 | 0.847 | 2.224 | 0.207 | 0.466 |
| GET_browse _call_numbers Z711 .C88 2013 | 5300 | 5300 | 0 | 0 | 8.572 | 0.091 | 0.262 | 0.387 | 0.647 | 1.157 | 2.992 | 0.318 | 0.647 |
call number browsing W/O background process
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers BT83.57 .N625 2013 | 2773 | 2773 | 0 | 0 | 7.757 | 0.047 | 0.144 | 0.223 | 0.378 | 0.778 | 2.292 | 0.181 | 0.378 |
| GET_browse _call_numbers CC76.4 .C66 2013 | 2772 | 2772 | 0 | 0 | 7.762 | 0.050 | 0.159 | 0.240 | 0.409 | 0.804 | 1.594 | 0.195 | 0.409 |
| GET_browse _call_numbers DT107.827.L32 1987 | 2774 | 2774 | 0 | 0 | 7.747 | 0.050 | 0.157 | 0.240 | 0.402 | 0.792 | 2.029 | 0.193 | 0.401 |
| GET_browse _call_numbers G3893.M5 1923 .W4 | 2770 | 2770 | 0 | 0 | 7.726 | 0.053 | 0.168 | 0.253 | 0.415 | 0.712 | 1.655 | 0.201 | 0.415 |
| GET_browse _call_numbers LC4019 .H83 2012 | 2773 | 2773 | 0 | 0 | 7.772 | 0.059 | 0.168 | 0.255 | 0.415 | 0.889 | 1.485 | 0.206 | 0.415 |
| GET_browse _call_numbers N6370 .C585 2013eb | 2774 | 2774 | 0 | 0 | 7.742 | 0.054 | 0.160 | 0.235 | 0.402 | 0.758 | 1.403 | 0.195 | 0.402 |
| GET_browse _call_numbers P302.77 .P67 2013 | 2773 | 2773 | 0 | 0 | 7.751 | 0.067 | 0.186 | 0.270 | 0.431 | 0.797 | 1.744 | 0.221 | 0.431 |
| GET_browse _call_numbers PR2274 .P7 1633 | 2771 | 2771 | 0 | 0 | 7.720 | 0.050 | 0.160 | 0.240 | 0.408 | 0.809 | 1.845 | 0.198 | 0.408 |
| GET_browse _call_numbers TL220 .B85 2012 | 2776 | 2776 | 0 | 0 | 7.736 | 0.041 | 0.140 | 0.213 | 0.381 | 0.816 | 1.294 | 0.175 | 0.381 |
| GET_browse _call_numbers Z711 .C88 2013 | 2774 | 2774 | 0 | 0 | 7.768 | 0.090 | 0.250 | 0.348 | 0.526 | 0.934 | 1.775 | 0.284 | 0.526 |
Average response times chart (blue- callnumber browsing random words, orange - call number browsing random words + CICO 10 users)_
it can be seen that the response time slightly (15%) increases when working with CICO in background.
long ramp-up period, 1 minute per user from 0 to 25 users
Call number browsing + DI + CICO
Call number browsing with old cluster
I user
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 2643 | 2643 | 0 | 0 | 2.233 | 0.044 | 0.125 | 0.164 | 0.285 | 0.494 | 0.948 | 0.145 | 0.278 |
10 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 25569 | 25569 | 0 | 0 | 21.363 | 0.043 | 0.138 | 0.194 | 0.350 | 0.611 | 1.937 | 0.167 | 0.346 |
20 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 41155 | 41155 | 0 | 0 | 35.026 | 0.041 | 0.196 | 0.355 | 0.671 | 1.078 | 2.625 | 0.271 | 0.669 |
50 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 103398 | 103398 | 0 | 0 | 89.437 | 0.041 | 0.211 | 0.313 | 0.534 | 0.896 | 2.146 | 0.251 | 0.530 |
100 users
| Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GET_browse _call_numbers | 106865 | 106865 | 0 | 0 | 93.836 | 0.044 | 0.727 | 0.978 | 1.394 | 2.073 | 4.713 | 0.758 | 1.389 |
