...

Summary

• During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
  
  • "memory": 1024 → 2304
  • "memoryReservation": 896 → 2048
  • -XX:MaxMetaspaceSize=128m → 256m
  • -Xmx768m → 1536m
• After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF ProfilesTests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  
• Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
• During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.

• No memory leaks, memory consumption was stable during all of the tests.

  Database CPU usage varied up to a maximum of 26%

Test Runs and Results

This table contains durations for all Workflows. 

Comparison

This graphs contains durations comparison between tests.

 Image Removed

...

This table contains Data Export durations for Test Set №2. 

...

This table contains Data Export durations for Test Set №3. 

...

This table contains Data Export durations for Test Set №4. 

...

This table contains Data Export durations for Test Set №5. 

...

Comparison

This graphs contains durations comparison between tests.

 Image Added

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 64% maximum.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 91%.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 42%.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 98%.

...

Inctanse CPU Utilization

Kafka metrics

DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

                                                                                                                     

Top SQL-queries

Resource utilization for Test №8

Service CPU Utilization

Here we can see that okapi used 38% CPU in average.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was maximum 53%.

DB Connections

Max number of DB connections was 3842.

DB load

                                                                                                                     

Top SQL-queries

...

PTF - Baseline MCPT environment configuration

• 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs
  db.r6g.4xlarge	128 GiB	16 vCPUs

  
  

• Open Search ptf-testtest 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...

• Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).
• Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
• Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
• Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
• Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
• Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
• Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
• Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.