Overview

• This document contains the results of testing MOBIUS master-script tests on the MCPT environment with different environment configurations.

Summary

• During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
  
  • "memory": 1024 → 2304
  • "memoryReservation": 896 → 2048
  • -XX:MaxMetaspaceSize=128m → 256m
  • -Xmx768m → 1536m
• After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.

• No memory leaks, memory consumption was stable during all of the tests.

• Database CPU usage varied up to a maximum of 26%

Test Runs and Results

This table contains Data Export durations for Test Set №1. 

...

This table contains Data Export durations for Test Set №2. 

...

This table contains Data Export durations for Test Set №3. 

...

Overview

• This document contains the results of testing MOBIUS master-script tests on the MCPT environment with different environment configurations.

Summary

• During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
  
  • "memory": 1024 → 2304
  • "memoryReservation": 896 → 2048
  • -XX:MaxMetaspaceSize=128m → 256m
  • -Xmx768m → 1536m
• After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.

• No memory leaks, memory consumption was stable during all of the tests.

• Database CPU usage varied up to a maximum of 26%

Test Runs and Results

This table contains Data Export durations for Test Set

...

№1. 

...

This table contains Data Export durations for Test Set №6№4. 

Service CPU Utilization

Here we can see that mod-data-export used 575% CPU in spike.

Service Memory Utilization

Here we can see that data-export module used 88% memory.

Kafka metrics

Image Removed

Image Removed

DB CPU Utilization

DB CPU was MAX 20-22% for all tests.

DB Connections

Max number of DB connections was 1500 in avarage.

DB load

Top SQL-queries

...

Service CPU Utilization

Here we can see that mod-data-export used 570% CPU in spike.

Service Memory Utilization

Here we can see that data-export module used 110% memory.

Kafka metrics

DB CPU Utilization

DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.

DB Connections

Max number of DB connections was 2230.

DB load

Top SQL-queries

...

Service CPU Utilization

Here we can see that mod-data-export used 230% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 108% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 20%.

DB Connections

Max number of DB connections was 2040.

DB load

Top SQL-queries

This table contains Data Export durations for Test Set №5. 

This table contains Data Export durations for Test Set №6. 

Service CPU Utilization

Here we can see that mod-data-export used 325% 575% CPU in spike.

Service Memory Utilization

Here we can see that mod-that data-export module used 73% 88% memory.

Kafka metrics

Image Added

Image Added

DB CPU Utilization

DB CPU was MAX 20-22% for all tests.

DB Connections

Max number of DB connections was 26301500 in avarage.

DB load

Top SQL-queries

Service CPU Utilization

Here we can see that mod-data-export used 130% 570% CPU in spike.

Service Memory Utilization

Here we can see that mod-datathat data-export module used 82% 110% memory.

Kafka metrics

DB CPU Utilization

DB CPU was 25%MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.

DB Connections

Max number of DB connections was 18502230.

DB load

Top SQL-queries

Service CPU Utilization

Here we can see that mod-data-export used 590% 230% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 82% 108% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%20%.

DB Connections

Max number of DB connections was 30502040.

DB load

Top SQL-queries

...

Appendix

Infrastructure

PTF -environment qcpt

...

1 database  instance, writer

...

db.r6g.xlarge

...

•  cs00000001_0042
  • instances - 637243
•  cs00000001
  • authorities - 55000
•  cs00000001_0003
  • holdings  - 146704
  • instances - 148713

...

• Data nodes
  • Instance type - r6g.2xlarge.search
  • Number of nodes - 4
  • Version: OpenSearch_2_7_R20240502
• Dedicated master nodes
  • Instance type - r6g.large.search
  • Number of nodes - 3

...

• 4 m5.2xlarge brokers in 2 zones

• Apache Kafka version 2.8.0

• EBS storage volume per broker 300 GiB

• auto.create.topics.enable=true
• log.retention.minutes=480
• default.replication.factor=3

...

Cluster Resources - qcpt-pvt (Fri Jul 26 09:55:21 UTC 2024)

Methodology/Approach

MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

• Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report:  task count 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).
• Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
• Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all modules.
• Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all modules.
• Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all modules.
• Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all modules.
• Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all modules.
• Test 8: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
• Test 11: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
• Test 12: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.

Test set №5:

• Test 13: Concurrently run Data Export using 500 instance records file with the Default instances export job profile, 500 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
• Test 14: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile, 1000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
• Test 15: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile, 2000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.

Test set №6:

• Test 16: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
• Test 17: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
• Test 18: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.

To get status and time range for import jobs the query used: 

...

Resource utilization for Test Set №4

Expand
title Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 325% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 73% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 22%.

DB Connections

Max number of DB connections was 2630.

DB load

Top SQL-queries

Service CPU Utilization

Here we can see that mod-data-export used 130% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

DB Connections

Max number of DB connections was 1850.

DB load

Top SQL-queries

Service CPU Utilization

Here we can see that mod-data-export used 590% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

DB Connections

Max number of DB connections was 3050.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF -environment qcpt

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• Number of records in DB:
  •  cs00000001_0042
    • instances - 637243
  •  cs00000001
    • authorities - 55000
  •  cs00000001_0003
    • holdings  - 146704
    • instances - 148713
• Open Search ptf-test
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology/Approach

MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

• Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).
• Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
• Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
• Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
• Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
• Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
• Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
• Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.