Overview

...

Overview

• Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances.  We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0.  To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments. 

...

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 64% maximum.

...

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Instance CPU Utilization

Image Added

Kafka metrics

DB CPU Utilization

DB CPU was 91%.

...

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Instance CPU Utilization

Image Added

Kafka metrics

DB CPU Utilization

DB CPU was 42%.

...

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

...

Instance CPU Utilization

Kafka metrics

DB CPU Utilization

DB CPU was 98%.

...

Inctanse CPU Utilization

Image Removed

Kafka metrics

Image Removed

Image Removed

DB CPU Utilization

DB CPU was 98%.

Image Removed

DB Connections

Max number of DB connections was 5150.

Image Removed

DB load

 The objective of this test is to evaluate the performance and behavior of the MCPT environment with a consistent instance type (r6g.xlarge), a fixed number of instances (14), and a new placement strategy ("one task per host"). By setting CPU=2 for all services, we aim to assess the impact of these changes on overall system performance, responsiveness, and stability, and to address the uneven CPU utilization observed in Test №6.

Results: The test revealed a minor performance improvement of approximately 3%-5% compared to Test №6. However, the total duration of all workflows showed a performance degradation of around 10%-15% compared to Baseline Test №1. The updated placement strategy of "one task per host" had a positive effect on addressing the previously observed imbalance but did not fully achieve the expected results. We still observed instances with high CPU loads (up to 80%) and others with significantly lower utilization (around 5%). Despite the more balanced distribution of tasks, some instances continued to have a high number of tasks (15+), while others had very few (2), leading to persistent CPU load imbalance.

Service CPU Utilization

Here we can see that okapi used 44000% of the unit CPU power.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Instance CPU Utilization

Image Added

Kafka metrics

Image Added

Image Added

DB CPU Utilization

DB CPU was 98%.

Image Added

DB Connections

Max number of DB connections was 5150.

Image Added

DB load

                                                                                                                     

Top SQL-queries

Image Removed

Resource utilization for Test №8

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Introduction: 

Objective: 

Results: 

                                                          

Top SQL-queries

Image Added

Test №8

Introduction:TheBaseline MCPT Environment configurationwas applied, theinstance typewas changed tox2gd.large, thenumber of instanceswas changed to14, placement strategy was updated to"one task per host",andCPU=2was set for allservices.

Objective: The objective of this test is to evaluate the performance and behavior of the MCPT environment using the x2gd.large instance type with a consistent number of instances (14) and the "one task per host" placement strategy. By setting CPU=2 for all services, we aim to assess the impact of these configurations on overall system performance, responsiveness, and stability, and to compare the results with previous tests to determine the effectiveness of the new setup.

Results: The test revealed a significant performance degradation, with the total average duration for all workflows being approximately 5 times worse compared to the Baseline Test №1. Additionally, there was an imbalance in CPU utilization across instances, indicating that some instances were underutilized while others were overloaded.

Service CPU Utilization

Here we can see that okapi used 38% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Instance CPU Utilization

Image Added

...

PTF - Baseline MCPT environment configuration

• 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs
  db.r6g.4xlarge	128 GiB	16 vCPUs

  
  

• Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
  • MSK 
  tenant
  
  • 4 m5.2xlarge ptf-KRaft-mode
    
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka

    version 2

    • version 3.

    8

    • 7.

    0

    • x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...

• Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).
• Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
• Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
• Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
• Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
• Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
• Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
• Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.