| Table of Contents |
|---|
Overview
- In this workflow, we perform a load test on mod-search for the Quesnelia release.
- Run these tests in both Quesnelia (qcp1) for baseline numbers and Eureka Quesnelia (qep1) environments and compare results
- Run test with 1, 20 virtual users on non-Eureka and Eureka environments.
Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-993
We tested with 1, 20 virtual users on non-Eureka and Eureka environments.
Summary
For a single user, performance is better in the Eureka environment. However, when scaled to 20 users, there is a performance degradation of approximately 7.8%
- Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 users - non-Eureka.
Service CPU utilization was up to 6% for mod-search and 5% for mod-inventory-storage, for all other services CPU did not exceed 4%.
Memory utilization was stable and no memory leaks or OOM issues were observed.
RDS CPU utilization was about up to 6%.
- Open search CPU utilization was up to 100% with 20 user tests.
...
Errors during 20 users - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.
Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.- For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
(8 VCPUs * 3) / 2 + 1 = 13 operations - For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
4 nodes * 13 = 52 operations
...
| transaction | Number Of Samples | Average | Median | pct90 | pct95 | pct99 | Min | Max |
|---|---|---|---|---|---|---|---|---|
| MSF_GET /search/authorities keyword <> random sentence | 3682 | 477 | 487 | 862 | 1148 | 2214 | 175 | 2400 |
| MSF_GET /search/authorities keyword <> randomword | 3684 | 462 | 472 | 850 | 1108 | 2423 | 166 | 2848 |
| MSF_GET /search/authorities keyword = *random sentence* | 3682 | 1545 | 1555 | 2153 | 2548 | 4332 | 941 | 4350 |
| MSF_GET /search/authorities keyword = random sentence | 3682 | 245 | 252 | 653 | 993 | 2623 | 19 | 2635 |
| MSF_GET /search/authorities keyword = randomword | 3684 | 234 | 236 | 608 | 973 | 3428 | 16 | 3451 |
| MSF_GET /search/authorities keyword == random sentence | 3683 | 225 | 233 | 591 | 885 | 1783 | 20 | 1792 |
| MSF_GET /search/authorities keyword == randomword | 3683 | 244 | 252 | 625 | 965 | 3221 | 16 | 3239 |
| MSF_GET /search/authorities keyword all *randomword* | 3683 | 1191 | 1206 | 1816 | 2117 | 4118 | 518 | 4139 |
| MSF_GET /search/authorities keyword all random sentence | 3683 | 243 | 251 | 631 | 936 | 3322 | 21 | 3341 |
| MSF_GET /search/authorities keyword all randomword | 3686 | 262 | 266 | 660 | 1035 | 2643 | 17 | 2652 |
| MSF_GET /search/authorities keyword any random sentence | 3683 | 272 | 279 | 603 | 902 | 2910 | 20 | 2920 |
| MSF_GET /search/authorities keyword any randomword | 3685 | 232 | 237 | 592 | 947 | 3309 | 16 | 3332 |
| MSF_GET /search/instances: contributors <> 2random words | 3679 | 408 | 418 | 786 | 1073 | 2685 | 133 | 2702 |
| MSF_GET /search/instances: contributors = *randomword | 3680 | 441 | 451 | 844 | 1064 | 2958 | 162 | 2969 |
| MSF_GET /search/instances: contributors = randomword* | 3680 | 227 | 234 | 561 | 882 | 3076 | 14 | 3094 |
| MSF_GET /search/instances: contributors == 2random words | 3679 | 217 | 226 | 594 | 902 | 2062 | 14 | 2073 |
| MSF_GET /search/instances: contributors all 2random words | 3677 | 231 | 239 | 613 | 952 | 3320 | 14 | 3338 |
| MSF_GET /search/instances: contributors all randomword | 3680 | 232 | 239 | 592 | 973 | 3142 | 14 | 3157 |
| MSF_GET /search/instances: contributors any 2random words | 3680 | 244 | 252 | 622 | 984 | 2020 | 15 | 2030 |
| MSF_GET /search/instances: keyword all random sentence AND languages == randomlang | 7354 | 233 | 237 | 614 | 901 | 3484 | 19 | 3511 |
| MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit") | 7350 | 346 | 356 | 772 | 1084 | 3477 | 39 | 3501 |
| MSF_GET /search/instances: languages == lang | 7349 | 344 | 350 | 797 | 1114 | 3435 | 38 | 3458 |
| MSF_GET /search/instances: languages == lang AND items.status.name == "Available" | 7350 | 353 | 360 | 775 | 1085 | 3426 | 39 | 3449 |
| MSF_GET /search/instances: subjects <> 1randomword | 3670 | 338 | 255 | 926 | 1177 | 2866 | 14 | 2880 |
| MSF_GET /search/instances: subjects = (randomword OR randomword) | 3682 | 247 | 256 | 609 | 918 | 2879 | 16 | 2896 |
| MSF_GET /search/instances: subjects = *1randomword | 3672 | 601 | 610 | 1032 | 1365 | 2939 | 268 | 2940 |
| MSF_GET /search/instances: subjects = 1randomword* | 3672 | 235 | 241 | 589 | 877 | 3489 | 14 | 3509 |
| MSF_GET /search/instances: subjects == 1randomword | 3670 | 232 | 238 | 595 | 1004 | 2564 | 15 | 2573 |
| MSF_GET /search/instances: subjects all 1randomword | 3672 | 241 | 248 | 635 | 898 | 3062 | 16 | 3082 |
| MSF_GET /search/instances: subjects all randomword AND title all randomword | 3682 | 237 | 243 | 620 | 976 | 2919 | 13 | 2935 |
| MSF_GET /search/instances: subjects all randomword NOT title all randomword | 3681 | 235 | 242 | 580 | 853 | 2493 | 16 | 2508 |
| MSF_GET /search/instances: subjects all randomword OR title all randomword | 3682 | 261 | 270 | 637 | 998 | 2581 | 26 | 2582 |
| MSF_GET /search/instances: subjects any 2random words | 3670 | 255 | 264 | 640 | 1031 | 2757 | 15 | 2759 |
| MSF_GET /search/instances: title <> 3 word sentence | 3670 | 415 | 424 | 770 | 1087 | 2284 | 140 | 2295 |
| MSF_GET /search/instances: title == 3 word sentence | 3670 | 237 | 244 | 634 | 1073 | 2361 | 18 | 2367 |
| MSF_GET /search/instances: title all *1randomword | 3670 | 3984 | 3988 | 5754 | 6226 | 7451 | 2260 | 7453 |
| MSF_GET /search/instances: title all 1randomword* | 3670 | 264 | 273 | 678 | 977 | 2829 | 17 | 2839 |
| MSF_GET /search/instances: title all 3 word sentence | 3670 | 236 | 243 | 628 | 891 | 3082 | 18 | 3091 |
| MSF_GET /search/instances: title any 3 word sentence | 3670 | 653 | 642 | 1974 | 2483 | 5288 | 36 | 5311 |
| MSF_TC: mod search by auth query fs09000000 | 3682 | 5627 | 5750 | 7399 | 8206 | 10533 | 2217 | 10551 |
| MSF_TC: mod search by boolean query fs09000000 | 3681 | 980 | 1015 | 1870 | 2341 | 4271 | 107 | 4272 |
| MSF_TC: mod search by contributors query fs09000000 | 3677 | 1998 | 2069 | 3160 | 3756 | 6881 | 423 | 6898 |
| MSF_TC: mod search by filter query fs09000000 | 3677 | 1267 | 1310 | 2360 | 2812 | 5026 | 144 | 5043 |
| MSF_TC: mod search by keyword query fs09000000 | 3672 | 1273 | 1315 | 2334 | 2785 | 6416 | 142 | 6444 |
| MSF_TC: mod search by subject query fs09000000 | 3670 | 1900 | 1881 | 3226 | 3637 | 5916 | 391 | 5920 |
| MSF_TC: mod search by title query fs09000000 | 3670 | 5787 | 5851 | 8198 | 9116 | 11512 | 2621 | 11530 |
Resource Utilization
Memory Utilization
1 user - non-Eureka
1 user - Eureka
20 users - non-Eureka
20 users - Eureka
Service CPU Utilization
1 user - non-Eureka
1 user - Eureka
...
20 users
...
Instance CPU Utilization
...
- non-Eureka
...
20 users -
...
Eureka
...
RDS CPU Utilization
1 user - non-Eureka
1 user - Eureka
20 users - non-Eureka
20 users - Eureka
RDS Connections count
1 user - non-Eureka
1 user - Eureka
20 users - non-Eureka
20 users - Eureka
OpenSearch metrics
Additional information
...
- Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000
- Artifact stored http://github.com/folio-org/perf-testing/mod-search
- runtime 60 minutes
...