Overview

• This document contains the results of testing MOBIUS master-script tests. Ticket:  PERF-809 - Getting issue details... STATUS

Summary

• All tests were run successfully and without errors on the Mobius-performance testing cluster (mcpt). Approximately over 100 tests with various environment configurations were conducted. The tests and configurations that showed the best results for two database modifications, such as db.r6g.4xlarge and db.r6g.8xlarge, will be demonstrated.
• During testing, an insufficient amount of resources for Amazon OpenSearch Service was detected, so data node instance type was increased to r6g.2xlarge.search.

• To improve performance, the AWS task count was adjusted for certain services, which increased the memory load on the database. To avoid OOM (Out of Memory) errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. Specifically, the shared_buffers parameter was modified to SUM({DBInstanceClassMemory/24076},-50003).

• To avoid "remaining connection slots are reserved for non-replication superuser and rds_superuser connections" errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. max_connections was modified to LEAST({DBInstanceClassMemory/9531392},9000)
•  For mod-authtoken CPU was changed from 512 to 1024 and task count was adjusted to 6 tasks (2 default), okapi and nginx-okapi CPU  was changed from 1024 to1536.

• No sign of memory leaks on every module during master-script tests. Memory shows a stable trend. For tests with a high load of single record create and update (62 virtual users without pauses) OOM issue of mod-inventory was reported  MODINV-1023 - Getting issue details... STATUS

• No modules are showing a growing CPU trend or any spikes. 

• Average response time differs and depends on environment configurations.

• The "Receiving: downloading" workflow. Due to its design, it's processed by the frontend and sends frequent requests to the database in a very large quantity. When under a load of 70% of tenants simultaneously, this causes significant degradation in the performance of all other workflows. Therefore, results are provided in tables with additional tests conducted with this workflow and without it. The CPU usage by the database increases by more than 25% with this workflow and reaches 99%.
• Database connection count did not exceed 5700
• During the high-load test, exporting 800,000 records concurrently for 62 tenants, an issue with the mod-data-export occurs PERF-927 - Getting issue details... STATUS , which will be resolved and tested in the next release - Quesnelia  PERF-888 - Getting issue details... STATUS . Other workflows performed normally. Resource usage was stable and corresponded to the load.
• Test with Fiscal year rollover average duration 3:21:51 for 30 tenants and 6:26:33 for 60 tenants, finished without errors, shows severe performance degradation for data import and data export workflows due to high database CPU usage 99.8%.

Recommendations

Database parameter group configuration:

Amazon OpenSearch Service

data node instance type - r6g.2xlarge.search

Services

mod-authtoken CPU - 1024, 6 tasks

okapi CPU-1536

nginx-okapi CPU-1536

For database size db.r6g.4xlarge

adjust task count to 6 for services:

• mod-authtoken

adjust task count to 4 for services:

• mod-permissions
• mod-search
• mod-patron
• mod-inventory
• mod-inventory-storage
• mod-circulation
• mod-circulation-storage
• mod-order
• mod-order-storage,
• mod-invoice
• mod-invoice-storage
• mod-users

For database size db.r6g.8xlarge

adjust task count to 6 for services:

• mod-authtoken
• mod-search

adjust task count to 4 for services:

• mod-permissions
• mod-patron
• mod-inventory
• mod-inventory-storage
• mod-circulation
• mod-circulation-storage
• mod-order
• mod-order-storage,
• mod-invoice
• mod-invoice-storage
• mod-organization
• mod-organization-storage
• mod-users
• mod-finance
• mod-finance-storage
• mod-configuration

"Receiving: downloading" workflow optimization

Test Runs 

Test Results

Test №1

Test №7 High load test

Test №8 Fiscal year rollover (30 tenants)

On the tenant cs00000001_0001 was performed test without other workflows and only 1 fiscal year rollover, duration was 1:24:52.

 * - for tenant cs00000001_0007 FYR was not started properly(mechanical mistake due to manual start). Database was restored from snapshot for the next test with 60 tenants and finished successful for all tenants.

Test №9 Fiscal year rollover (60 tenants)

Comparison of the test results with tuned and not tuned environment

The following diagram compares all test results of MOBIUS Baseline Tests: CI/CO.

Service CPU Utilization

It did not exceed 95% for all of the modules.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 98% average with ERW: Exporting Receiving Information to CSV and 75% without.

DB Connections

DB connections maximum 5700.

Resource utilization for Test #4

Service CPU Utilization

It did not exceed 95% for all of the modules.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 38% average.

DB Connections

DB connections were up to 5420.

Resource utilization for Test #5

Service CPU Utilization

It did not exceed 95% for all of the modules.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

DB CPU Utilization

DB CPU was 38% average.

DB Connections

DB connections were up to 5210.

Resource utilization for Test #6 longevity

Service CPU Utilization

Test completed successfully without sudden fails and/or modules restart. All modules are stable.

CPU utilisation didn't exceed 90%. (most used module is mod-permissions). 

Only module that has spikes in CPU usage is mod-pubsub. (may be related to background jobs)

Service Memory Utilization

No memory leaks found. 

All modules behaviour is stable.

DB CPU Utilization

DB Connections

Db used ±6K connections during whole duration of a test

Resource utilization for Test #7 High load

Instance CPU Utilization

CPU utilization didn't exceed 55%. 

Service CPU Utilization

CPU utilization didn't exceed 110%. 

Service Memory Utilization

It was observed an issue for mod-data-export PERF-927 - Getting issue details... STATUS . 

All modules' behavior is stable except mod-data-export.

DB CPU Utilization

DB Connections

Db used up to 7K connections

Open Search Resources utilization

Kafka Resources utilization

Resource utilization for Test #8 FYR (30 tenants)

Instance CPU Utilization

It did not exceed 26%.

Service CPU Utilization

It did not exceed 85% for all of the modules.

Service Memory Utilization

All modules' behavior is stable.

DB CPU Utilization

DB CPU was 99.8% average.

DB Connections

Db used up to 5K connections

DB Freeable Memory

During the test, it was not less than 35G of freeable memory.

Resource utilization for Test #9 FYR (60 tenants)

Instance CPU Utilization

It did not exceed 27%.

Service CPU Utilization

It did not exceed 100% for all of the modules.

Service Memory Utilization

All modules' behavior is stable.

DB CPU Utilization

DB CPU was 99.8% average.

DB Connections

Db used up to 5K connections

Appendix

Infrastructure

PTF - environment mcpt

• 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance, writer

  one of

  Name	Memory GIB	vCPUs
  db.r6g.4xlarge	128 GiB	16 vCPUs
  db.r6g.8xlarge	256 GiB	32 vCPUs

• Amazon OpenSearch Service: ptf-test
  
  • Data nodes
    Instance type-r6g.2xlarge.search
    Number of nodes-4
  • Dedicated master nodes
    Instance type-r6g.large.search
    Number of nodes-3
• MSK ptf-mobius-testing2
  
  • 2 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2

Modules memory and CPU parameters:

db.r6g.4xlarge

Methodology/Approach

MOBIUS Tests: scenarios were started by carrier.io or from load generator instance in the same (us-east-1) AWS region. 

The database contains five times more generated data for large tenant IDs compared to small tenant IDs. The table below displays the average count of generated instances and items for both small and large tenants..

The attached file contains full info about generated data per tenant.

• large tenants IDs = 0042, 0024, 0051, 0006

1. Run data preparation script before test
2. Create artifacts and upload them to carrier-io or to load generator instance in the same (us-east-1) AWS region
3. Change parameters and run tests
4. Executed one workflow per minute concurrently (2 for 4 large tenants).

Discussions