Skip to end of banner
Go to start of banner

PTF - Performance testing of Consolidating EC2 Instances (mcpt)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 39 Next »

Overview

  • Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances.  We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0.  To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments. 

PERF-942 - Getting issue details... STATUS  

Summary

  • Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
  • During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.

  • No memory leaks, memory consumption was stable during all of the tests.

  • Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

Test Runs and Results

This table contains durations for all Workflows. 

WorkflowTest 1
Baseline Env
configuration		Test 2
CPU=0		Test 3
x2gd.xlarge
CPU=0
instances=6		Test 4
x2gd.xlarge
CPU=0
instances=8		Test 5
x2gd.large
CPU=0
instances=10		Test 6
r6g.xlarge
CPU=2
instances=12(14) 		Test 7
r6g.xlarge
CPU=2
instances=14
placement strategy
(one task per host)		Test 8
x2gd.large
CPU=2
instances=14
placement strategy
(one task per host)
Average
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		Errors
AIE_TC: Create Invoices7682100%8106100%50468100%15554100%92805100%10191100%9716100%91611100%
AIE_TC: Invoices Approve2818100%3016100%24781100%7720100%51554100%4512100%4210100%51626100%
AIE_TC: Paying Invoices2882100%3164100%32994100%9788100%64642100%5430100%4902100%65029100%
CICO_TC_Check-In Controller20000%22140%190750%60380%329090%36590%31940%308310%
CICO_TC_Check-Out Controller35480%37990%292241%105600%573470%64390%55950%555030%
CSI_TC:Share local instance1305019%1307319%1426516%161920%173612%1378214%1333917%174601%
DE_Exporting MARC Bib records custom workflow847070%590990%38249156%1168690%56738983%1278850%885410%50075971%
DE_Exporting MARC Bib records workflow741260%430930%41798257%976564%62361091%821260%830460%45721258%
EVA_TC: View Account5190%5650%1603419%17360%406192%11381%9721%376691%
ILR_TC: Create ILR14220%14990%116071%38820%236760%24140%19970%223350%
MSF_TC: mod search by auth query7552%6680%18880%15000%61800%10800%9100%58940%
MSF_TC: mod search by boolean query2051%1590%5940%4270%20220%2580%2380%18710%
MSF_TC: mod search by contributors4401%3940%8560%8390%34540%6160%5300%33000%
MSF_TC: mod search by filter query3020%2840%5120%5390%20180%4160%3620%19240%
MSF_TC: mod search by keyword query3100%2800%5210%5370%20130%4160%3610%19120%
MSF_TC: mod search by subject query4480%4060%7970%7780%30760%6200%5190%29170%
MSF_TC: mod search by title query10901%10250%14350%13870%36870%14320%11490%35430%
OPIH_/oai/records53300%54040%93300%76770%68810%33270%69900%83350%
POO_TC: Add Order Lines521420%541930%2821920%792060%4120040%577350%577490%3999170%
POO_TC: Approve Order406560%425230%2117470%564460%2651670%439300%438340%2759350%
POO_TC Create Order307340%316520%1073180%429400%492340%321210%438340%1756430%
RTAC_TC: edge-rtac37350%38280%162950%13870%571950%42050%39500%555950%
SDIC_Single Record Import (Create)1327919%1389419%4502416%173050%790532%1465014%1453117%742011%
SDIU_Single Record Import (Update)184660%194320%218270100%282070%1183990%217360%209650%1157770%
TC: Receiving-an-Order-Line43765100%46104100%218270100%65242100%325230100%49024100%48538100%322267100%
Serials-Receiving-Workflow45694100%47336100%198116100%68545100%302203100%49873100%50028100%295508100%
Unreceiving-a-Piece7823100%7757100%40059100%13335100%64155100%9018100%8717100%60295100%
ULR_TC: Users loan Renewal Transaction28100%30780%226021%78290%383830%46730%40300%361430%

Comparison

This graph shows the total durations of all workflows compared between tests.

 


Test №1

Introduction: Baseline MCPT Environment configuration.

Objective: The goal of this test is to establish a baseline performance benchmark for the MCPT environment under standard conditions. This test will provide a reference to measure the impact of any subsequent changes, such as modifications in CPU allocation or other resource management strategies. The baseline will be used to assess the performance, responsiveness, and stability of the system with default configurations.

Service CPU Utilization

Here we can see that CPU did not exceed 95% of the unit CPU power for all of the modules.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

DB Connections

Max number of DB connections was 5568 in maximum.



Test №2

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The goal of this test is to evaluate the performance and behavior of the MCPT environment when CPU reservations are set to 0 for all modules. By not reserving any specific CPU resources, we aim to observe how the system dynamically allocates CPU resources based on availability and demand, and to assess any impacts on module performance, responsiveness, and overall system stability.

Results: Some workflows exhibited significant performance improvements, and the overall average duration for all workflows was better than in Baseline Test №1.

Service CPU Utilization

Here we can see that okapi used 13% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

DB Connections

Max number of DB connections was 5788.

DB load

                                                                                                                     

Top SQL-queries


Test №3

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.

Objective: The objective of this test is to assess the impact of using fewer, more memory-optimized instances (from m6g.2xlarge to x2gd.xlarge) on the performance of the MCPT environment. By reducing the number of instances while selecting a different instance type with a higher memory-to-vCPU ratio, this test aims to observe how the system handles workloads under these conditions and whether the overall efficiency and performance improve.

Results: This configuration led to a significant performance degradation, with performance being four times worse compared to the Baseline Test №1. The reduced number of instances in this setup was clearly insufficient to maintain the required performance levels.

Service CPU Utilization

Here we can see that mod-permissions used 20% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 64% maximum.

DB Connections

Max number of DB connections was 2040.


Test №4

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.

Objective: The goal of this test is to evaluate whether increasing the number of instances from 6 to 8, while maintaining the instance type as x2gd.xlarge and setting CPU=0 for all services, can mitigate the performance degradation observed in Test №3. This test aims to determine the optimal instance count needed to maintain stable performance in the MCPT environment under these conditions

Results: As a result, there was a 3x improvement in overall average duration for all workflows compared to Test №3. However, performance still lagged by more than 30% when compared to the Baseline Test №1.

Service CPU Utilization

Here we can see that okapi used 20% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics




DB CPU Utilization

DB CPU was 91%.

DB Connections

Max number of DB connections was 4840.

DB load

                                                                                                                    

Top SQL-queries


Test №5

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.

Objective: The goal of this test is to evaluate the impact of switching to a smaller instance type, x2gd.large, with a slightly increased instance count of 10, while keeping CPU reservations at 0 for all services. This test aims to find a balance between instance size and count that could potentially optimize performance while reducing costs. We seek to observe how these changes affect the overall system performance compared to previous tests.

Results: The test resulted in a significant performance degradation, with a 7x reduction in performance compared to Baseline Test №1. The change to the x2gd.large instance type and the increased number of instances did not mitigate the performance issues, leading to a much worse overall performance compared to previous configurations in Test №4.

Service CPU Utilization

Here we can see that okapi used 36% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


DB CPU Utilization

DB CPU was 42%.

DB Connections

Max number of DB connections was 3650.

DB load

                                                                                                                    

Top SQL-queries


Test №6

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.

Objective: To assess the performance and behavior of the MCPT environment with a different instance type (r6g.xlarge) and increased CPU allocation (CPU=2 for all services), while using a similar number of instances (12 out of 14). The goal is to determine the impact of these changes on overall system performance, responsiveness, and stability compared to the Baseline configuration.

Results: The test revealed a minor performance degradation of approximately 15-20% in the total duration of all workflows compared to Baseline Test №1. Additionally, we observed uneven distribution of CPU utilization across instances. Some instances experienced high CPU loads of up to 80%, while others had significantly lower utilization, around 5%. This imbalance appears to be due to two main factors: firstly, some modules had only a few tasks concentrated on single instances, and secondly, there were instances with 15 or more tasks, while others had only 0-2 tasks.

Service CPU Utilization

Here we can see that okapi used 46000% CPU of unit power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Inctanse CPU Utilization

Kafka metrics




DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

                                                                                                                     

Top SQL-queries


Test №7

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Objective: The objective of this test is to evaluate the performance and behavior of the MCPT environment with a consistent instance type (r6g.xlarge), a fixed number of instances (14), and a new placement strategy ("one task per host"). By setting CPU=2 for all services, we aim to assess the impact of these changes on overall system performance, responsiveness, and stability, and to address the uneven CPU utilization observed in Test №6.

Results: The test revealed a minor performance improvement of approximately 3%-5% compared to Test №6. However, the total duration of all workflows showed a performance degradation of around 10%-15% compared to Baseline Test №1. The updated placement strategy of "one task per host" had a positive effect on addressing the previously observed imbalance but did not fully achieve the expected results. We still observed instances with high CPU loads (up to 80%) and others with significantly lower utilization (around 5%). Despite the more balanced distribution of tasks, some instances continued to have a high number of tasks (15+), while others had very few (2), leading to persistent CPU load imbalance.

Service CPU Utilization

Here we can see that okapi used 44000% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Inctanse CPU Utilization

Kafka metrics




DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

                                                                                                                     

Top SQL-queries


Resource utilization for Test №8

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Introduction: 

Objective: 

Results: 

Service CPU Utilization

Here we can see that okapi used 38% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was maximum 53%.

DB Connections

Max number of DB connections was 3842.

DB load

                                                                                                                     

Top SQL-queries


Appendix

Infrastructure

PTF - Baseline MCPT environment configuration

  • 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameMemory GIBvCPUs

    db.r6g.4xlarge

    		128 GiB16 vCPUs
  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


 mcpt modules memory and CPU parameters

Cluster Resources - mcpt-pvt 

ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
mod-remote-storage18mod-remote-storage:3.0.124920447210243960512512
mod-ncip12mod-ncip:1.14.42102489612876888128
mod-finance-storage14mod-finance-storage:8.5.041024896102470088128
mod-agreements17mod-agreements:6.0.2015921488128000
mod-ebsconet12mod-ebsconet:2.1.1212481024128700128256
edge-sip212edge-sip2:3.1.12102489612876888128
mod-consortia18mod-consortia:1.0.325136477610244416384512
mod-organizations12mod-organizations:1.8.02102489612876888128
mod-settings17mod-settings:1.0.22102489620076888128
edge-dematic15edge-dematic:2.1.11102489612876888128
mod-data-import15mod-data-import:3.0.71204818442561292384512
mod-search18mod-search:3.0.5425922480204814405121024
mod-tags12mod-tags:2.1.02102489612876888128
mod-authtoken22mod-authtoken:2.14.1614401152102492288128
edge-courses12edge-courses:1.3.02102489612876888128
edge-inventory6edge-inventory:1.4.02102489612876888128
mod-inventory-update12mod-inventory-update:3.2.12102489612876888128
mod-notify12mod-notify:3.1.02102489612876888128
mod-configuration13mod-configuration:5.9.24102489612876888128
mod-orders-storage13mod-orders-storage:13.6.04102489651270088128
edge-caiasoft12edge-caiasoft:2.1.02102489612876888128
mod-login-saml15mod-login-saml:2.7.22102489612876888128
mod-erm-usage-harvester7mod-erm-usage-harvester:4.4.12102489612876888128
mod-password-validator12mod-password-validator:3.1.0214401298128768384512
mod-gobi13mod-gobi:2.7.12102489612876888128
mod-licenses15mod-licenses:5.0.22248023121281792384512
mod-fqm-manager12mod-fqm-manager:1.0.32102489612876888128
edge-dcb9edge-dcb:1.0.22102489612876888128
mod-bulk-operations18mod-bulk-operations:1.1.923072260010241536384512
mod-graphql12mod-graphql:1.12.00102489612876888128
mod-finance12mod-finance:4.8.04102489612876888128
mod-erm-usage7mod-erm-usage:4.6.02102489612876888128
mod-lists15mod-lists:1.0.52102489612876888128
mod-copycat12mod-copycat:1.5.02102489612876888128
mod-permissions26mod-permissions:6.4.04168415445121024384512
mod-entities-links17mod-entities-links:2.0.4225922480400144001024
pub-edge6pub-edge:2023.06.142102489612876800
mod-orders13mod-orders:12.7.142048144010241024384512
edge-patron14edge-patron:5.0.0.1582102489625676888128
edge-ncip12edge-ncip:1.9.22102489612876888128
mod-users-bl12mod-users-bl:7.6.021440115251292288128
edge-ea-data-export7edge-ea-data-export:4.1.02102489612876888128
mod-invoice13mod-invoice:5.7.241440115251292288128
mod-inventory-storage19mod-inventory-storage:27.0.444096369020483076384512
mod-user-import12mod-user-import:3.8.02102489612876888128
mod-sender12mod-sender:1.11.02102489612876888128
edge-oai-pmh15edge-oai-pmh:2.7.121512136010241440384512
mod-data-export-worker15mod-data-export-worker:3.1.223072280010242048384512
mod-rtac12mod-rtac:3.5.02102489612876888128
mod-task-list7mod-task-list:1.9.22102489612876888128
mod-circulation-storage20mod-circulation-storage:17.1.742880259215361814384512
mod-calendar13mod-calendar:2.5.02102489612876888128
mod-source-record-storage21mod-source-record-storage:5.7.525600500020483500384512
mod-event-config12mod-event-config:2.6.02102489612876888128
mod-courses12mod-courses:1.4.82102489612876888128
mod-circulation-item9mod-circulation-item:1.0.02102489612876888128
mod-inventory19mod-inventory:20.1.642880259210241814384512
mod-email12mod-email:1.16.02102489612876888128
mod-circulation20mod-circulation:24.0.1142880259215361814384512
mod-pubsub16mod-pubsub:2.11.32153614401024922384512
mod-di-converter-storage17mod-di-converter-storage:2.1.52102489612876888128
edge-rtac15edge-rtac:2.6.22102489612876888128
edge-orders15edge-orders:2.9.12102489612876888128
mod-template-engine12mod-template-engine:1.19.12102489612876888128
mod-users17mod-users:19.2.24102489612876888128
mod-patron-blocks14mod-patron-blocks:1.9.021024896102476888128
mod-audit13mod-audit:2.8.02102489612876888128
edge-fqm13edge-fqm:1.0.12102489612876888128
mod-source-record-manager18mod-source-record-manager:3.7.725600500020483500384512
nginx-edge6nginx-edge:2023.06.1421024896128000
mod-quick-marc16mod-quick-marc:5.0.11228821761281664384512
nginx-okapi9nginx-okapi:2023.06.14210248961536000
okapi-b8okapi:5.1.23168414401536922384512
mod-feesfines12mod-feesfines:19.0.02102489612876888128
mod-invoice-storage13mod-invoice-storage:5.7.041872153610241024384512
edge-users6edge-users:1.2.02102489612876888128
mod-dcb9mod-dcb:1.0.02102489612876888128
mod-service-interaction15mod-service-interaction:3.0.22204818442561290384512
mod-data-export17mod-data-export:4.8.711024896102476888128
mod-patron12mod-patron:6.0.04102489612876888128
mod-oai-pmh16mod-oai-pmh:3.12.824096369020483076384512
edge-connexion12edge-connexion:1.1.12102489612876888128
mod-kb-ebsco-java12mod-kb-ebsco-java:4.0.02102489612876888128
mod-notes13mod-notes:5.1.021024896128952384512
mod-data-export-spring15mod-data-export-spring:3.0.21204818442561536384512
mod-login12mod-login:7.10.12144012981024768384512
mod-organizations-storage13mod-organizations-storage:4.6.04102489612876888128
pub-okapi6pub-okapi:2023.06.142102489612876800
edge-erm7edge-erm:1.0.02102489612876888128
mod-eusage-reports7mod-eusage-reports:2.0.02102489612876888128


Methodology/Approach

MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

  • Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).
  • Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
  • Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
  • Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
  • Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
  • Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
  • Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
  • Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.


  • No labels