Overview

...

• For a single user, performance is better in the Eureka environment. However, when scaled to 20 users, there is a performance degradation of approximately 7.8%

• Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 users - non-Eureka. 

• Service CPU utilization was up to 60% 6% for mod-search and 5% for mod-inventory-storage, for all other services CPU did not exceed 4%.

• Memory utilization was stable and no memory leaks or OOM issues were observed.

• RDS CPU utilization was about up to 6%.

• Open search CPU utilization was up to 100% with 20 user tests.

Recommendations & Jiras

Errors during 20 users - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

• Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
• For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
  (8 VCPUs * 3) / 2 + 1 = 13 operations
• For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
  4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

...

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

...

60 minutes tests

Test#	Test name	results	Comments
1	1 user - non-Eureka	Success	0 errors
2	20 users - non-Eureka	Completed with errors	3224 errors out of 199179 API calls ERROR: "Too Many Requests"
3	20 users - non-Eureka

(avg)20 users - Eureka (avg)%Total requests processed in 60 minutes test run19657208115.9%199179183864-7.6%Average throughput5.44 ops/s5.77 ops/s6%

55.2 ops/s

50.9 ops/s-7.8%Average response time194185-4.6%3653824.7%

retest	Completed with errors	41894 errors out of 252249 API calls ERROR: "Too Many Requests"
4	1 user - Eureka	Success	0 errors
5	20 users - Eureka	Success	0 errors

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

Test Details

...

1 user - non-Eureka

...

Resource Utilization

Memory Utilization

1 user - non-Eureka

1 user - Eureka

20 users - non-Eureka

20 users - Eureka

Service CPU Utilization

1 user - non-Eureka

1 user - Eureka

20 users - non-Eureka

20 users - Eureka

Instance CPU Utilization

1 user - non-Eureka

1 user - Eureka

20 users - non-Eureka

20 users - Eureka

RDS CPU Utilization

1 user - non-Eureka

1 user - Eureka

20 users - non-Eureka

20 users - Eureka

RDS Connections count

1 user - non-Eureka

1 user - Eureka

20 users - non-Eureka

20 users - Eureka

OpenSearch metrics

Additional information

...

• Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000 

• Artifact stored http://github.com/folio-org/perf-testing/mod-search
• runtime 60 minutes

...