Skip to end of banner
Go to start of banner

PTF - Performance testing of Consolidating Instances (mcpt)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Overview

  • This document contains the results of Data Export tests on 62 tenants (excluding the central tenant) on the QCPT environment, the previous report is available. 

PERF-942 - Getting issue details... STATUS  

Summary

  • During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
    • "memory": 1024 → 2304
    • "memoryReservation": 896 → 2048
    • -XX:MaxMetaspaceSize=128m → 256m
    • -Xmx768m → 1536m
  • After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.

  • No memory leaks, memory consumption was stable during all of the tests.

  • Database CPU usage varied up to a maximum of 26%

Test Runs and Results

This table contains Data Export durations for Test Set №1. 


500
(instance records)		1000
(instance records)		2000
(instance records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss
Duration
hh:mm:ss
Member tenants 75pct0:00:010:00:030:00:05
Member tenants MAX0:00:040:00:040:00:08
cs00000001_0042
(maximum instance records)		0:07:550:07:320:07:41
 Full results table for Test Set №1



This table contains Data Export durations for Test Set №2. 


500
(instance records)		1000
(instance records)		2000
(instance records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss
Duration
hh:mm:ss
Member tenants 75pct0:00:020:00:050:00:16
Member tenants MAX0:00:030:00:050:00:20
cs00000001_0042
(maximum instance records)		0:06:120:06:220:06:09
 Full results table for Test Set №2

This table contains Data Export durations for Test Set №3. 


500
(instance records)		1000
(instance records)		2000
(instance records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss
Duration
hh:mm:ss
Member tenants 75pct0:00:010:00:020:00:04
Member tenants MAX0:00:010:00:020:00:05
 Full results table for Test Set №3

This table contains Data Export durations for Test Set №4. 


500
(instance records)		1000
(instance records)		2000
(instance records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss
Duration
hh:mm:ss
Member tenants 75pct0:00:010:00:020:00:04
Member tenants MAX0:00:020:00:030:00:05
cs00000001
(maximum authority records)		0:01:360:01:440:02:05
 Full results table for Test Set №4

This table contains Data Export durations for Test Set №5. 


500
(records)		1000
(records)		2000
(records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss		Duration
hh:mm:ss
cs00000001_0003
(instance records)		0:00:030:00:040:00:07

cs00000001_0003
(holdings records)

0:00:010:00:010:00:02
cs00000001
(maximum authority records)		0:02:180:01:490:01:44
cs00000001_0003
(maximum holdings records)		0:02:110:01:130:01:14

This table contains Data Export durations for Test Set №6. 


500
(instance records)		1000
(instance records)		2000
(instance records)
TenantDuration
hh:mm:ss		Duration
hh:mm:ss		Duration
hh:mm:ss
cs00000001_00420:00:080:00:070:00:12
cs00000001_0042
(maximum instance records)		0:06:210:05:410:05:41

Resource utilization for Test Set №1

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 575% CPU in spike.


Service Memory Utilization

Here we can see that data-export module used 88% memory.



Kafka metrics



DB CPU Utilization

DB CPU was MAX 20-22% for all tests.


DB Connections

Max number of DB connections was 1500 in avarage.


DB load

Top SQL-queries



Resource utilization for Test Set №2

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 570% CPU in spike.


Service Memory Utilization

Here we can see that data-export module used 110% memory.



Kafka metrics




DB CPU Utilization

DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.


DB Connections

Max number of DB connections was 2230.


DB load

Top SQL-queries



Resource utilization for Test Set №3

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 230% CPU.


Service Memory Utilization

Here we can see that mod-data-export used 108% memory .



Kafka metrics





DB CPU Utilization

DB CPU was 20%.


DB Connections

Max number of DB connections was 2040.


DB load

Top SQL-queries



Resource utilization for Test Set №4

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 325% CPU.


Service Memory Utilization

Here we can see that mod-data-export used 73% memory .



Kafka metrics





DB CPU Utilization

DB CPU was 22%.


DB Connections

Max number of DB connections was 2630.


DB load

Top SQL-queries



Resource utilization for Test Set №5

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 130% CPU.


Service Memory Utilization

Here we can see that mod-data-export used 82% memory .



Kafka metrics





DB CPU Utilization

DB CPU was 25%.


DB Connections

Max number of DB connections was 1850.


DB load

Top SQL-queries



Resource utilization for Test Set №6

 Resource utilization table

Service CPU Utilization

Here we can see that mod-data-export used 590% CPU.


Service Memory Utilization

Here we can see that mod-data-export used 82% memory .



Kafka metrics





DB CPU Utilization

DB CPU was 25%.


DB Connections

Max number of DB connections was 3050.


DB load

Top SQL-queries



Appendix

Infrastructure

PTF -environment qcpt

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    		32 GiB4 vCPUs2731
  • Number of records in DB:
    •  cs00000001_0042
      • instances - 637243
    •  cs00000001
      • authorities - 55000
    •  cs00000001_0003
      • holdings  - 146704
      • instances - 148713
  • Open Search ptf-test
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


 qcpt modules memory and CPU parameters

Cluster Resources - qcpt-pvt (Fri Jul 26 09:55:21 UTC 2024)


Methodology/Approach

Test №1:

  • MOBIUS Tests: scenarios were started by JMeter script from load generator
  • Baseline MCPT Environment configuration according  to previous report:  task count 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536

Test set №2:

  • Test 4: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • Test 5: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • Test 6: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.

Test set №3:

  • Test 7: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.
  • Test 8: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.
  • Test 9: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.

Test set №4:

  • Test 10: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
  • Test 11: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
  • Test 12: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.

Test set №5:

  • Test 13: Concurrently run Data Export using 500 instance records file with the Default instances export job profile, 500 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
  • Test 14: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile, 1000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
  • Test 15: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile, 2000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.

Test set №6:

  • Test 16: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
  • Test 17: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
  • Test 18: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.

To get status and time range for import jobs the query used: 

 SQL Query


  • No labels