Overview

1. In this testing effort we perform load tests of mod-search on the Quesnelia release.
2. These tests with 1, 20 virtual users were run in both Okapi Quesnelia (qcp1) environment for baseline numbers and Eureka Quesnelia (qecp1) environment with the goal of comparing Eureka's performance against Okapi FOLIO. 
3. PERF-993 - Getting issue details... STATUS

Summary

• For a single user, performance is better in the Eureka environment. However, when scaled to 20 vusers, there is a performance degradation of approximately 7.8%

• It is impossible to reach an average throughput higher than 50.9 ops/s for the Eureka environment (for non-Eureka environment over 55.2 ops/s). A higher number of users up to 60 will only increase response time even if we have 4 tasks for mod-search. We observed an increase in response time from sidecar-mod-search from 282ms for 5 vusers and up to 1,2 seconds for 60 vusers. Additional information in this table.
• Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 vusers - non-Eureka. 
• Open search CPU utilization was up to 100% with 20 user tests.

• Service CPU utilization was up to 14% for mod-search, for all other services CPU did not exceed 6%.

• Memory utilization was stable and no memory leaks or OOM issues were observed.

• RDS CPU utilization was about up to 10% for all tests.

• Non-Eureka environment uses 50 fewer database connections than Eureka. The number of connections for mod-search was the same for both environments

                  {
                      "name": "DB_MAXPOOLSIZE",
                      "value": "20"
                  },

Recommendations & Jiras

Errors during 20 vusers - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

• Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
• For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
  (8 VCPUs * 3) / 2 + 1 = 13 operations
• For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
  4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

60 minutes tests

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

Test Details

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka