PTF - Performance Instance Resources Optimization - QCP1

Owned by Stanislav Nehrii
Last updated: Sep 25, 2024

Overview

  • The primary objective of testing was to evaluate the performance of the Baseline MCPT Environment configuration while attempting to optimize costs by adjusting instance types and reducing the number of instances. The tests were designed to compare the performance outcomes across different configurations, including variations in instance types and counts within multiple Auto Scaling Groups (ASGs). By systematically modifying these variables, the goal was to maintain or improve the performance observed in the baseline configuration while achieving cost efficiency.

PERF-962 - Getting issue details... STATUS  

Summary

  • Through a series of experiments involving different placement strategies, instance types, and total instance counts, we found that the performance remained consistent when using these configurations: 
    • three c7g.large instances dedicated to the okapi service alongside five r7g.2xlarge instances for all other services, with the CPU parameter set to 2 for all services.
    • five r7g.2xlarge instances for all services, with the CPU parameter set to 2 for all services.
  • Optimized environment configurations offers a 20-40% cost reduction compared to the existing setup, making it a more economical option without compromising on performance.
  • Configurations with three c7g.large instances for the okapi service and five r7g.2xlarge instances for all other services show the best performance across all experiments.
  • In fact, some workflows show better performance with this new setup than correct infrastructures.
  • The CPU utilization on EC2 level better now - around 30-60%, previously it was under 20%.

AWS Configuration Costs

ClusterInstance TypeCost per Month
(USD)		Number of InstancesTotal Cost per Cluster
(USD)
QCP1 m6g.2xlarge$221.7610$2,217.60
MCPT m6g.2xlarge$221.7614$3,104.64
Optimized Infrastructure
Two Auto Scaling Groups		c7g.large$52.203$1,698.84
r7g.2xlarge$308.455
Optimized Infrastructure
One Auto Scaling Groups		r7g.2xlarge$308.455$1,542.25


Cost Comparison (Before vs After)

Cluster

Previous Total Cost
(USD)

New Total Cost
(USD)

Percentage Saving
(%)

QCP1$2,217.60$1,698.8423.39%
MCPT$3,104.64$1,698.8445.28%

Test Runs

Test #DescriptionStatus
Test 1Instance type: m6g.2xlarge. Instances count: 10. Completed
Test 2Instance type: m6g.2xlarge. Instances count: 10 (Repeat Test 1). Completed
Test 3Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services.Completed
Test 4Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services (Repeat Test 3).Completed
Test 5CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules.Completed
Test 6CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules (Repeat Test 5).Completed
Test 7CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules.Completed
Test 8

CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services.


Test 9CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services (Repeat Test 8).
Test 10CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services.
Test 11

CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services (Repeat Test 10).


Test 12CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services (Repeat Test 11).

Test Results

This table contains durations for all Workflows. 

WorkflowsTest 1

Test 2
(Repeat 1)

Test 3Test 4
(Repeat 3)		Test 5 Test 6
(Repeat 5)		Test 7Test 8Test 9
(Repeat 8)		Test 10Test 11
(Repeat 10)		Test 12
(Repeat 11)

Average response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		ErrorsAverage response time
(milliseconds)		Errors
DATA IMPORT0:52:03
0:44:55
0:46:07
0:47:09
0:51:41
0:58:35
1:00:03
0:43:53
0:47:06
0:55:30
0:45:46
0:44:09
DATA EXPORT0:58:11
0:44:43
0:47:41
0:50:32
0:38:59
0:45:53
0:48:26not finished for main0:45:41
0:48:49
0:56:49
0:42:16
0:44:26
CICO_TC_Check-In Controller11630%9480%9320%9580%8490%8950%9400%9120%9930%11760%8920%9670%
CICO_TC_Check-Out Controller16970%14810%14080%14280%13180%13180%13670%13450%14450%16750%13710%14670%
DE_Exporting MARC Bib records workflow25280%38180%36750%28300%19180%22230%18650%33630%24200%18720%33980%50330%
ILR_TC: Create ILR10230%8740%7300%8040%6240%6620%8020%8200%7670%11160%8770%7800%
ILR_TC: Get ItemId1320%1310%1070%1140%1120%1090%1090%1160%1270%1650%1360%1300%
MSF_TC: mod search by auth query48300%48356%54806%765810%757019%21750%24740%609410%411912%29380%554611%707518%
MSF_TC: mod search by boolean query4690%6213%14562%10275%18765%2561%6050%13435%5274%3330%11977%12636%
MSF_TC: mod search by contributors4750%16553%18056%17143%34678%4960%5030%19664%7556%4650%20905%20368%
MSF_TC: mod search by filter query2290%8442%7032%11370%15732%2170%2620%11002%4293%2380%9444%10686%
MSF_TC: mod search by keyword query2280%5193%11614%12458%13124%1980%2560%11443%4342%2500%9794%10188%
MSF_TC: mod search by subject query5770%18463%18253%16347%23282%4450%5390%21633%8407%4730%14766%19198%
MSF_TC: mod search by title query21410%32513%37701%45982%48878%20380%22940%42855%32146%20540%41457%44379%
DI_TC: Importing MARC records workflow Transaction &{tenant}10594510%204230%9356260%9504030%10478360%179460%121042933%172580%9553490%22382080%189740%187620%
PRV_TC: View Patron record Group3470%3100%2480%2380%2130%2840%3130%2650%2540%3550%2420%2730%
ULR_TC: Users loan Renewal Transaction10594510%18520%15280%17170%15510%15950%17490%15260%17610%41170%14960%16170%


Comparison

This graph shows the durations of all workflows compared between the best test results.

Part 1

 

Part 2

Part 3



Test №1

Introduction: Test 1: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run. 

Objective: The objective of test was to evaluate the performance of the MCPT environment by applying the baseline configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-inventory-b modules used 73% CPU.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


OpenSearch metrics


DB CPU Utilization

DB CPU was 97% average.

DB Connections

Max number of DB connections was 1250 in maximum.

DB load

                                                                                                                    

Top SQL-queries

Test №2

Introduction: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run (Repeat Test 1).

Objective: The objective of this test was to validate the consistency of performance observed in Test 1 by repeating the same configuration.

Results: Results was almost the same for all workflows.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-inventory-b modules used 102% CPU in maximum.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


OpenSearch metrics

DB CPU Utilization

DB CPU was 95% average 

DB Connections

Max number of DB connections was 1250.

DB load

                                                                                                                    

Top SQL-queries

Test №3-4

Introduction: Test 3: The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run. Test 3: Repeat Test 1.

Objective: The objective of test was to evaluate the performance of the QCP1 environment by applying two distinct autoscaling groups with different instance types: c7g.large for the Okapi service and r7g.xlarge for all other services. By running test, the goal was to determine if this modified configuration could achieve similar or improved performance compared to the baseline, while potentially optimizing resource allocation and cost. 

Results: We observed nearly identical performance results almost for all the workflows compared to the Baseline configuration and repeated it in Test №4.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-inventory-b used 78k% of the CPU.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics

OpenSearch metrics

DB CPU Utilization

DB CPU was 97% maximum.

DB Connections

Max number of DB connections was 2200.

DB load

                                                                                                                    

Top SQL-queries

Test №5-6-7

Introduction: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run.

Objective: Repeat tests to confirm performance.

Results: Performance was confirmed at the same like baseline.

Instance CPU Utilization


Service CPU Utilization

Here we can see that mod-data-export-b used 82k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


OpenSearch metrics

DB CPU Utilization

DB CPU was 97%.

DB Connections

Max number of DB connections was 1250.

DB load

                                                                                                                    

Top SQL-queries

Test №8-9

Introduction:  The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run.

Objective: The goal of test is to put okapi service in one tier with all services and compare performance with previous result and baseline result.

Results: Performance was confirmed at the same like baseline.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


OpenSearch metrics


DB CPU Utilization

DB CPU was 94%.

DB Connections

Max number of DB connections was 1265.

DB load

                                                                                                                    

Top SQL-queries


Test №10

Introduction:  The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results: Performance was confirmed at the same like previous one.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-data-export-b used 116k% MAX of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


OpenSearch metrics


DB CPU Utilization

DB CPU was 94%.

DB Connections

Max number of DB connections was 1220.

DB load

                                                                                                                    

Top SQL-queries


Test №11

Introduction:  The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results: Performance was confirmed at the same like previous one.

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


OpenSearch metrics


DB CPU Utilization

DB CPU was 94%.

DB Connections

Max number of DB connections was 1200.

DB load

                                                                                                                    

Top SQL-queries


Test №12

Introduction:  The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results: Performance was confirmed at the same like previous one.

Instance CPU Utilization


Service CPU Utilization

Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


OpenSearch metrics


DB CPU Utilization

DB CPU was 94%.

DB Connections

Max number of DB connections was 1265.

DB load

                                                                                                                    

Top SQL-queries

Appendix

Infrastructure

PTF - Baseline QCP1 environment configuration (was changed during testing)

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameMemory GIBvCPUs

    db.r6g.xlarge

    		32 GB4 vCPUs
  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK fse-tenant
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


 qcp1 modules memory and CPU parameters

Cluster Resources - qcp1-pvt (Thu Sep 12 09:30:08 UTC 2024)

ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage6mod-remote-storage:3.2.024920447203960512512
mod-ncip6mod-ncip:1.14.421024896076888128
mod-finance-storage6mod-finance-storage:8.6.021024896070088128
mod-agreements6mod-agreements:7.0.02159214880000
mod-ebsconet6mod-ebsconet:2.2.02124810240700128256
mod-organizations6mod-organizations:1.9.021024896076888128
mod-consortia4mod-consortia:1.1.025136477604416384512
edge-sip24edge-sip2:3.2.0-SNAPSHOT.20921024896076888128
mod-serials-management6mod-serials-management:1.0.022480231201792384512
mod-settings6mod-settings:1.0.321024896076888128
mod-data-import9mod-data-import:3.1.012048184401292384512
edge-dematic6edge-dematic:2.2.011024896076888128
mod-search6mod-search:3.2.0225922480014405121024
mod-inn-reach4mod-inn-reach:3.2.0-SNAPSHOT.86236003240028805121024
mod-tags6mod-tags:2.2.021024896076888128
edge-courses6edge-courses:1.4.021024896076888128
mod-authtoken10mod-authtoken:2.15.1214401152092288128
mod-inventory-update6mod-inventory-update:3.3.021024896076888128
mod-notify6mod-notify:3.2.021024896076888128
mod-configuration6mod-configuration:5.10.021024896076888128
mod-orders-storage6mod-orders-storage:13.7.021024896070088128
edge-caiasoft6edge-caiasoft:2.2.021024896076888128
mod-login-saml6mod-login-saml:2.8.021024896076888128
mod-erm-usage-harvester6mod-erm-usage-harvester:4.5.021024896076888128
mod-licenses6mod-licenses:6.0.022480231201792384512
mod-gobi6mod-gobi:2.8.021024896076888128
mod-password-validator6mod-password-validator:3.2.02144012980768384512
mod-bulk-operations6mod-bulk-operations:2.0.023072260001536384512
mod-fqm-manager6mod-fqm-manager:2.0.121024896076888128
edge-dcb6edge-dcb:1.1.021024896076888128
mod-graphql6mod-graphql:1.12.12102489612876888128
mod-finance6mod-finance:4.9.021024896076888128
mod-erm-usage6mod-erm-usage:4.7.022800255001800384512
mod-batch-print7mod-batch-print:1.1.021024896076888128
mod-copycat6mod-copycat:1.6.021024896076888128
mod-lists6mod-lists:2.0.021024896076888128
mod-entities-links7mod-entities-links:3.0.02259224800144001024
mod-permissions11mod-permissions:6.5.021684154401024384512
pub-edge4pub-edge:2023.06.1421024896076800
mod-orders6mod-orders:12.8.022048174001024384512
edge-patron3edge-patron:5.1.02102489625676888128
edge-ncip6edge-ncip:1.9.221024896076888128
mod-marc-migrations2mod-marc-migrations:1.0.0-SNAPSHOT.50204817641024153688128
edge-inn-reach4edge-inn-reach:3.1.1-SNAPSHOT.4521024896076888128
mod-users-bl6mod-users-bl:7.7.0214401152092288128
mod-oa5mod-oa:2.1.0-SNAPSHOT.6221024896076888256
mod-inventory-storage3mod-inventory-storage:27.1.024096369020483076384512
mod-invoice8mod-invoice:5.8.1214401152092288128
mod-user-import6mod-user-import:3.8.021024896076888128
mod-sender7mod-sender:1.12.021024896076888128
edge-oai-pmh6edge-oai-pmh:2.9.021512136001440384512
mod-data-export-worker6mod-data-export-worker:3.2.123072280002048384512
mod-rtac3mod-rtac:3.6.02102489612876888128
mod-circulation-storage3mod-circulation-storage:17.2.022880259215361814384512
mod-calendar6mod-calendar:3.1.021024896076888128
mod-source-record-storage18mod-source-record-storage:5.8.725600500020483500384512
mod-event-config6mod-event-config:2.7.021024896076888128
mod-courses6mod-courses:1.4.1021024896076888128
mod-circulation-item6mod-circulation-item:1.0.021024896076888128
mod-inventory3mod-inventory:20.2.022880259210241814384512
mod-email6mod-email:1.17.022800255001800384512
mod-pubsub6mod-pubsub:2.13.02153614400922384512
mod-circulation3mod-circulation:24.2.022880259215361814384512
mod-di-converter-storage6mod-di-converter-storage:2.2.021024896076888128
edge-rtac3edge-rtac:2.7.12102489612876888128
edge-orders6edge-orders:3.0.021024896076888128
mod-users7mod-users:19.3.121024896076888128
mod-template-engine6mod-template-engine:1.20.021024896076888128
mod-patron-blocks6mod-patron-blocks:1.10.021024896076888128
mod-audit6mod-audit:2.9.021024896076888128
edge-fqm6edge-fqm:2.0.021024896076888128
mod-source-record-manager7mod-source-record-manager:3.9.0-SNAPSHOT.33025600500003500384512
nginx-edge4nginx-edge:2023.06.14210248960000
mod-quick-marc8mod-quick-marc:5.1.012288217601664384512
nginx-okapi4nginx-okapi:2023.06.14210248960000
okapi-b6okapi:5.3.03168414400922384512
mod-feesfines6mod-feesfines:19.1.021024896076888128
mod-invoice-storage6mod-invoice-storage:5.8.021872153601024384512
mod-dcb7mod-dcb:1.1.021024896076888128
mod-service-interaction6mod-service-interaction:4.0.122048184401290384512
mod-data-export19mod-data-export:5.0.41204818440000
mod-patron3mod-patron:6.1.02102489612876888128
mod-oai-pmh6mod-oai-pmh:3.13.024096369003076384512
edge-connexion6edge-connexion:1.2.021024896076888128
mod-kb-ebsco-java6mod-kb-ebsco-java:4.0.021024896076888128
mod-notes6mod-notes:5.2.0210248960952384512
mod-data-export-spring6mod-data-export-spring:3.2.012048184401536384512
mod-organizations-storage6mod-organizations-storage:4.7.021024896076888128
mod-login6mod-login:7.11.02144012980768384512
pub-okapi4pub-okapi:2023.06.1421024896076800
mod-eusage-reports6mod-eusage-reports:2.1.121024896076888128


Methodology/Approach

Tests scenarios were started by JMeter script from load generator.   

Baseline QCP1 Environment configuration: Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage,  number of tasks to launch for service mod-marc-migrations-b was set zero. Instance type: m6g.2xlarge. Instances count: 10. Database db.r6g.xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).

  • Test 1: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run.
  • Test 2: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run (Repeat Test 1).
  • Test 3:  The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run.
  • Test 4: The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run (Repeat Test 3).
  • Test 5: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run.
  • Test 6: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run (Repeat Test 5).
  • Test 7: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type:  c7g.large for others services, qcp1_MiniMaster.jmx script was run.
  • Test 8: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run.
  • Test 9: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run (Repeat Test 8).
  • Test 10: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run.
  • Test 11: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run (Repeat Test 10).
  • Test 12: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type:  c7g.large for all services, qcp1_MiniMaster.jmx script was run (Repeat Test 11).