Content Comparison

...

• During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
  
  • "memory": 1024 → 2304
  • "memoryReservation": 896 → 2048
  • -XX:MaxMetaspaceSize=128m → 256m
  • -Xmx768m → 1536m
• After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.

• No memory leaks, memory consumption was stable during all of the tests.

• Database CPU usage varied up to a maximum of 26%
• During the tests we had a some background process on Database side

...

Here we can see that data-export module used 88% memory.

Kafka metrics

DB CPU Utilization

DB CPU was MAX 20-22% for all tests.

...

Here we can see that data-export module used 110% memory.

Kafka metrics

DB CPU Utilization

DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.

...

Here we can see that mod-data-export used 108% memory .

Kafka metrics

DB CPU Utilization

DB CPU was MAX 20% for Test 3.

DB Connections

Max number of DB connections was 2040.

...

Here we can see that mod-data-export used 73% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 22%.

...

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

...

Here we can see that mod-data-export used 82% memory .

Kafka metrics

DB CPU Utilization

DB CPU was 25%.

...

Infrastructure

PTF -environment qcpt

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge4xlarge	32 128 GiB	4 vCPUs	273116 vCPU

  
  

• Number of records in DB:
  •  cs00000001_0042
    • instances - 637243
  •  cs00000001
    • authorities - 55000
  •  cs00000001_0003
    • holdings  - 146704
    • instances - 148713
• Open Search ptf-test
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...