Overview

• In this report, PTF conducts performance testing to identify the appropriate environment scale that will ensure the fulfillment of the NFRs outlined in ASA NFRs. The goal is also to identify potential bottlenecks in the system that may require code changes to optimize performance and meet the specified requirements. PERF-1079 - Getting issue details... STATUS  

Summary

• During testing with different configurations, we observed that once OpenSearch exceeded 80% of its SPU capacity, performance was significantly impacted, leading to degraded ASA operations. As a result, we determined that the ASA load with additional traffic on the PTF-LOC environment should not exceed 80% of OpenSearch's SPU capacity. Therefore, we will use the r7g.4xlarge instance type to maintain optimal performance.

• During ASA testing with extra load on the PTF-LOC environment, OpenSearch SPU utilization reached 62%, compared to 52% without the extra load. We expected that OpenSearch CPU utilization would increase to 70% with the additional load, as in Test 1, where 31 sessions were lost due to insufficient capacity.

• Additionally, we found that operations involving the "CONTAINS_ALL" search criterion were significantly slower, with response times 6 to 12 times higher compared to other criteria.

• Despite the extra load on the PTF-LOC environment, overall performance remained stable, with variations of less than 5%.

• NFR_007 - Performance degradation should be less than 10% for each block of active users, which meets the NFR requirements.

• NFR_006 - The 95th percentile of response time is greater than 1 second, which does not match the NFR due to the use of the "CONTAINS_ALL" search criterion. However, the 99th percentile of response time is less than 2 seconds, which meets the NFR, even with "CONTAINS_ALL" included.

• NFR_004 - The 95th percentile of response time exceeds 2 seconds for some "CONTAINS_ALL" operations, which does not match the NFR, but all other ASA operations match the NFR.

• The total number of searching and viewing operations is 31 per second. Since we lost 31 user sessions, we can infer that the operations are close to 33 per second, which meets the NFR requirements.

Test Runs

Test Results

This table contains response times and percentage degradation for API Load Testing with a Gradual Increase of Users Per Block tests.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi and that mod-search-ebsco modules used around 1.6% of CPU Instance power.

Service Memory Utilization

Here we can see that all modules show a stable trend.

Kafka metrics

OpenSearch metrics

Data Nodes CPU utilization

Data Nodes JVM memory pressure

Search rate

Search latency

JVM Search thread pool

DB CPU Utilization

DB CPU was 70% in spike.

DB Connections

Max number of DB connections was 2010.

Appendix

Infrastructure

PTF - QCP1 environment configuration (was changed during testing)

• 5-6 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GB	4 vCPUs

• Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK fse-tenant
  
  • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x 

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

PTF - eureka environment Ramsons (relc)

• 12 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance, writer

  Name	Memory GIB	vCPUs
  db.r7g.xlarge	32 GiB	4 vCPUs

• MSK fse-test
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2

Methodology/Approach

CICO Tests scenarios were started for 100 users and concurrently RTAC for 10 users by JMeter script from load generator. 

QCP1 configuration and steps to configure ECS infrastructure:

• Instance type r7g.2xlarge

• When New Relic/Open Telementry enabled OTEL value in mod-inventory's task definition set to true (OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED=true)

• mod-oa-b service was turn off for all tests
• during testing CPU value was applied for list of modules:
  • mod-inventory: 3072
  • mod-inventory-storage: 2048
  • mod-circulation: 1536
  • mod-circulation-storage: 1536
  • mod-feesfines: 256
  • mod-orders: 1024
  • mod-orders-storage: 512
  • mod-login: 1024
  • mod-source-record-storage: 2048
  • mod-rtac: 128
  • mod-patron: 128

Test 1:
The QCP1 environment was configured with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned ON, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 2:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test artifacts: