Overview

1. In this testing effort we perform load tests of mod-search on the Quesnelia release.
2. These tests with 1, 20 virtual users were run in both Okapi Quesnelia (qcp1) environment for baseline numbers and Eureka Quesnelia (qecp1) environment with the goal of comparing Eureka's performance against Okapi FOLIO. 
3. PERF-993 - Getting issue details... STATUS

Summary

• For a single user, performance is better in the Eureka environment. However, when scaled to 20 vusers, there is a performance degradation of approximately 7.8%

• It is impossible to reach an average throughput higher than 50.9 ops/s for the Eureka environment (for non-Eureka environment over 55.2 ops/s). A higher number of users up to 60 will only increase response time even if we have 4 tasks for mod-search. We observed an increase in response time from sidecar-mod-search from 282ms for 5 vusers and up to 1,2 seconds for 60 vusers. Additional information in this table.
• Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 vusers - non-Eureka. 
• Open search CPU utilization was up to 100% with 20 user tests.

• Service CPU utilization was up to 14% for mod-search, for all other services CPU did not exceed 6%.

• Memory utilization was stable and no memory leaks or OOM issues were observed.

• RDS CPU utilization was about up to 10% for all tests.

• Non-Eureka environment uses 50 fewer database connections than Eureka. The number of connections for mod-search was the same for both environments

                  {
                      "name": "DB_MAXPOOLSIZE",
                      "value": "20"
                  },

Recommendations & Jiras

Errors during 20 vusers - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

• Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
• For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
  (8 VCPUs * 3) / 2 + 1 = 13 operations
• For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
  4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

60 minutes tests

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

Test Details

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

Resource Utilization

Memory Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

Service CPU Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

RDS CPU Utilization

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

RDS Connections count

1 vuser - non-Eureka

1 vuser - Eureka

20 vusers - non-Eureka

20 vusers - Eureka

OpenSearch metrics

Appendix

Infrastructure

Files

Modules

PTF -  QCP1 environment configuration

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GB	4 vCPUs

• MSK fse-tenant
  
  • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x 

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
•  Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_13_R20240520-P5
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3

Additional information

environments: qcp1-non-Eureka, qecp1-Eureka

• Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000 

• Artifact stored http://github.com/folio-org/perf-testing/mod-search
• test performed for 1 vuser ramp-up 1 sec
• test performed for 20 vusers ramp-up 200 sec
• runtime 60 minutes

Run mod-search load testing for the following variant of queries:

• authority key search

• boolean search

• contributors search

• filter search

• keyword search

• subjects search
• title search
• wildcard search
• wildcard prefix search

Additional screenshots:

They show how number of users affects sidecars response time (data from New Relic):

start of the test up to 5 vusers

10 vusers

20 vusers

30 vusers

40 vusers