...

• All tests were run successfully and without errors on the Mobius-performance testing cluster (mcpt). Approximately over 100 tests with various environment configurations were conducted. The tests and configurations that showed the best results for two database modifications, such as db.r6g.4xlarge and db.r6g.8xlarge, will be demonstrated.
• During testing, an insufficient amount of resources for Amazon OpenSearch Service was detected, so data node instance type was increased to r6g.2xlarge.search.

• To improve performance, the AWS task count was adjusted for certain services, which increased the memory load on the database. To avoid OOM (Out of Memory) errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. Specifically, the shared_buffers parameter was modified to SUM({DBInstanceClassMemory/24076},-50003).

• To avoid "remaining connection slots are reserved for non-replication superuser and rds_superuser connections" errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. max_connections was modified to LEAST({DBInstanceClassMemory/9531392},9000)
•  For mod-authtoken CPU was changed from 512 to 1024 and task count was adjusted to 6 tasks (2 default), okapi and nginx-okapi CPU  was changed from 1024 to1536.

• No sign of memory leaks on every module during master-script tests. Memory shows a stable trend. For tests with a high load of single record create and update (62 virtual users without pauses) OOM issue of mod-inventory was reported 

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODINV-1023

• No modules are showing a growing CPU trend or any spikes. 

• Average response time differs and depends on environment configurations.

• The "Receiving: downloading" workflow. Due to its design, it's processed by the frontend and sends frequent requests to the database in a very large quantity. When under a load of 70% of tenants simultaneously, this causes significant degradation in the performance of all other workflows. Therefore, results are provided in tables with additional tests conducted with this workflow and without it. The CPU usage by the database increases by more than 25% with this workflow and reaches 99%.
• Database connection count did not exceed 5700
• During the high-load test, exporting 800,000 records concurrently for 62 tenants, an issue with the mod-data-export occurs

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-927

  , which will be resolved and tested in the next release - Quesnelia 

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-888

• Test with Fiscal year rollover for 60 tenants shows severe performance degradation for data import and data export workflows due to high database CPU usage 99.8%

Recommendations

Database parameter group configuration:

Amazon OpenSearch Service

data node instance type - r6g.2xlarge.search

Services

mod-authtoken CPU - 1024, 6 tasks

okapi CPU-1536

nginx-okapi CPU-1536

For database size db.r6g.4xlarge

adjust task count to 6 for services:

• mod-authtoken

adjust task count to 4 for services:

• mod-permissions
• mod-search
• mod-patron
• mod-inventory
• mod-inventory-storage
• mod-circulation
• mod-circulation-storage
• mod-order
• mod-order-storage,
• mod-invoice
• mod-invoice-storage
• mod-users

For database size db.r6g.8xlarge

adjust task count to 6 for services:

• mod-authtoken
• mod-search

adjust task count to 4 for services:

• mod-permissions
• mod-patron
• mod-inventory
• mod-inventory-storage
• mod-circulation
• mod-circulation-storage
• mod-order
• mod-order-storage,
• mod-invoice
• mod-invoice-storage
• mod-organization
• mod-organization-storage
• mod-users
• mod-finance
• mod-finance-storage
• mod-configuration

"Receiving: downloading" workflow optimization

Test Runs 

...

Test №8 Fiscal year rollover (30 tenants)

On the tenant cs00000001_0001 was performed test without other workflows and only 1 fiscal year rollover, duration was 1:24:52.

Test №9 Fiscal year rollover (60 tenants)

...

...

 * - for tenant cs00000001_0007 FYR was not started properly(mechanical mistake due to manual start). Database was restored from snapshot for the next test with 60 tenants and finished successful for all tenants.

Test №9 Fiscal year rollover (60 tenants)

...

PTF - environment mcpt

• 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance, writer

  one of

  Name	Memory GIB	vCPUs
  db.r6g.4xlarge	128 GiB	16 vCPUs
  db.r6g.8xlarge	256 GiB	32 vCPUs

  
  

• Amazon OpenSearch Service: ptf-test
  
  • Data nodes
    Instance type-r6g.2xlarge.search
    Number of nodes-4
  • Dedicated master nodes
    Instance type-r6g.large.search
    Number of nodes-3
• MSK ptf-mobius-testing2
  
  • 2 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2

...

Modules memory and CPU parameters:

db.r6g.4xlarge

...

MOBIUS Tests: scenarios were started by carrier.io or from load generator instance in the same (us-east-1) AWS region. 

The database contains five times more generated data for large tenant IDs compared to small tenant IDs. The table below displays the average count of generated instances and items for both small and large tenants..

The attached file contains full info about generated data per tenant.

• large tenants IDs = 0042, 0024, 0051, 0006

...