Table of Contentsoutline

Overview

• This document contains the results of Data Export tests on 62 tenants (excluding the central tenant) on the QCPT environment, the previous report is available. 

...

• During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
  
  • "memory": 1024 → 2304
  • "memoryReservation": 896 → 2048
  • -XX:MaxMetaspaceSize=128m → 256m
  • -Xmx768m → 1536m
• After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.

• No memory leaks, memory consumption was stable during all of the tests.

• Database CPU usage varied up to a maximum of 26%

...

Here we can see that data-export module used 88% memory.

Kafka metrics

DB CPU Utilization

DB CPU was MAX 20-22% for all tests.

...

Here we can see that data-export module used 110% memory.

Kafka metrics

DB CPU Utilization

DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.

...

Here we can see that mod-data-export used 108% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 20%.

...

Here we can see that mod-data-export used 73% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 22%.

...

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

...

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

...

Infrastructure

PTF -environment qcpt

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• Number of records in DB:
  •  cs00000001_0042
    • instances - 637243
  •  cs00000001
    • authorities - 55000
  •  cs00000001_0003
    • holdings  - 146704
    • instances - 148713
• Open Search ptf-test
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...

Cluster Resources - qcpt-pvt (Fri Jul 26 09:55:21 UTC 2024)

Methodology/Approach

Test set №1:

• Test 1: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter. 
• Test 2: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
• Test 3: Concurrently run Data Export using 200 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • .factor=3

Methodology/Approach

Test №1:

• MOBIUS Tests: scenarios were started by JMeter script from load generator
• Baseline MCPT Environment configuration according  to previous report:  task count 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536

Test set №2:

• Test 4: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
• Test 5: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
• Test 6: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.

...