Overview

1. In this workflow, we perform a load test on mod-search for the Quesnelia release.
2. Run these tests in both Quesnelia (qcp1) for baseline numbers and Eureka Quesnelia (qep1) environments and compare results 
3. Run test with 1, 20 virtual users on non-Eureka and Eureka environments. 
4. PERF-993 - Getting issue details... STATUS

Summary

• For a single user, performance is better in the Eureka environment. However, when scaled to 20 vusers, there is a performance degradation of approximately 7.8%

• Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 vusers - non-Eureka. 

• Service CPU utilization was up to 14% for mod-search, for all other services CPU did not exceed 6%.

• Memory utilization was stable and no memory leaks or OOM issues were observed.

• RDS CPU utilization was about up to 10% for all tests.

• Non-Eureka environment uses 50 fewer database connections than Eureka.
• Open search CPU utilization was up to 100% with 20 user tests.

Recommendations & Jiras

Errors during 20 vusers - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

• Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
• For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
  (8 VCPUs * 3) / 2 + 1 = 13 operations
• For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
  4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

60 minutes tests

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

Test Details

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

Resource Utilization

Memory Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

Service CPU Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

RDS CPU Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

RDS Connections count

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

OpenSearch metrics

Appendix

Infrastructure

Files

Modules

PTF -  QCP1 environment configuration

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GB	4 vCPUs

• Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK fse-tenant
  
  • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x 

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Additional information

environments: qcp1-non-Eureka, qecp1-Eureka

• Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000 

• Artifact stored http://github.com/folio-org/perf-testing/mod-search
• test performed for 1 vuser ramp-up 1 sec
• test performed for 20 vusers ramp-up 200 sec
• runtime 60 minutes

Run mod-search load testing for the following variant of queries:

• authority key search

• boolean search

• contributors search

• filter search

• keyword search

• subjects search
• title search
• wildcard search
• wildcard prefix search