Versions Compared

Version Old Version 1 New Version Current
Changes made by

Stanislav Nehrii

Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

Overview

  • Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances.  We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0.  To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments. 

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-942
 

Summary

  • Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
  • During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.

  • No memory leaks, memory consumption was stable during all of the tests.

  • Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

Test Runs and Results

This table contains durations for all Workflows. 

...

Table of Contents
outlinetrue

Overview

  • In this report, PTF investigates the impact of setting CPU allocations to 0 units across all tasks within an AWS ECS cluster. The purpose of this study is to determine whether removing CPU constraints reveals the actual CPU usage of the tasks and to assess how this adjustment affects overall performance. By comparing key workflows across different environments, we aim to identify any potential changes in efficiency, throughput, or resource utilization that may result from setting CPU = 0. The findings from these tests will help inform best practices for resource allocation and performance optimization within ECS clusters.

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-959
 

Summary

  • During the tests Parameter CPU=0 improved performance for a several workflows, like MSF_TC: mod search by auth query, MSF_TC: mod search by contributors, MSF_TC: mod search by title query and ULR_TC: Users loan Renewal Transaction.

  • No memory leaks, memory consumption was stable during all of the tests.

  • Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

Test Runs

Test #DescriptionStatus
Test 1The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (average case) MOBIUS test was run.Completed
Test 2The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (high load case) MOBIUS test was run.Completed
Test 3The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (average case) MOBIUS test was run (rerun Test 1).Completed
Test 4The Baseline MCPT Environment configuration was applied, and CPU=1 was set for all modules,  Fixed Load (average case) MOBIUS test was run.Completed
Test 5The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.Completed
Test 6The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules,  Data Import - Create with 25k and 100k records files tests were run.Completed
Test 7The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.Completed
Test 8The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules,  Data Import - Create with 25k and 100k records files tests were run.Completed
Test 9The Baseline QCP1 Environment configuration was applied, and CPU=0 was set for all modules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.Completed
Test 10The Baseline QCP1 Environment configuration was applied, and CPU=0 was set for all modules,  Data Import - Create with 25k and 100k records files tests were run.Completed

Test Results

This table contains durations for all Workflows. 

Workflow

Test 1
(average case)

CPU=0

Test 2
(high load)

CPU=0

Test 3
(average case) 

CPU=0

Test 4
(average case) 

CPU=1


Average
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		ErrorsAverage
(milliseconds)		Errors
AIE_TC: Create Invoices8350100%13458100%7421100%8251100%
AIE_TC: Invoices Approve3180100%5251100%2960100%3088100%
AIE_TC: Paying Invoices3289100%4919100%3060100%3285100%
CICO_TC_Check-In Controller23230%33227%22280%23130%
CICO_TC_Check-Out Controller41540%570010%38960%39850%
CSI_TC:Share local instance1300819%1478114%1296920%1302219%
DE_Exporting MARC Bib records custom workflow529830%37991295%787590%1007000%
DE_Exporting MARC Bib records workflow427850%40202698%739570%1097940%
EVA_TC: View Account8033%9623%7001%8323%
ILR_TC: Create ILR15270%22154%14510%15180%
MSF_TC: mod search by auth query
7552%668
6720%
1888
1830
0%
7%
1500
3096
0%61800%10800%9100%5894
4%10230%
MSF_TC: mod search by boolean query
2051%1590%5940%4270%2022
1650%
258
485
0%
2%
238
737
0%
1%
1871
2120%
MSF_TC: mod search by contributors
440
398
1%
0%
394
1063
0%
3%
856
1664
0%
2%
839
6040%
3454
MSF_TC: mod search by filter query2860%
616
713
0%
2%
530
1070
0%
2%
3300
4170%
MSF_TC: mod search by
filter
keyword query
302
2840%
284
658
0%
1%
512
1006
0%
1%
539
4220%
2018
MSF_TC: mod search by subject query4070%
416
1112
0%
1%
362
1530
0%
2%
1924
6230%
MSF_TC: mod search by
keyword
title query
310
10310%
2800%521
24491%29071%17580%
5370%2013
OPIH_/oai/records60420%
416
4649
0%
100%
361
35870%
1912
54480%
MSF
POO_TC:
mod search by subject query448
Add Order Lines553340%
406
99076
0%
19%
797
544400%
778
552240%
3076
POO_TC: Approve Order425670%
620
79022
0%
12%
519
421910%
2917
429070%
MSF
POO_TC
: mod search by title query10901%10250%1435
Create Order319330%5646813%314060%
1387
321660%
3687
RTAC_TC: edge-rtac41500%
1432
47260%
1149
40840%
3543
40990%
OPIH_/oai/records53300%54040%93300%76770%68810%33270%69900%83350%POO_TC: Add Order Lines521420%541930%2821920%792060%4120040%577350%577490%3999170%POO_TC: Approve Order406560%425230%2117470%564460%2651670%439300%438340%2759350%POO_TC Create Order307340%316520%1073180%429400%492340%321210%438340%1756430%RTAC_TC: edge-rtac37350%38280%162950%13870%571950%42050%39500%555950%SDIC_Single Record Import (Create)1327919%1389419%4502416%173050%790532%1465014%1453117%742011%SDIU_Single Record Import (Update)184660%194320%218270100%282070%1183990%217360%209650%1157770%TC: Receiving-an-Order-Line43765100%46104100%218270100%65242100%325230100%49024100%48538100%322267100%Serials-Receiving-Workflow45694100%47336100%198116100%68545100%302203100%49873100%50028100%295508100%Unreceiving-a-Piece7823100%7757100%40059100%13335100%64155100%9018100%8717100%60295100%ULR_TC: Users loan Renewal Transaction28100%30780%226021%78290%383830%46730%40300%361430%

Comparison

This graph shows the total durations of all workflows compared between tests.

 

Resource utilization for Test №1

Baseline MCPT Environment configuration

SDIC_Single Record Import (Create)1377719%2047513%1377320%1387319%
SDIU_Single Record Import (Update)195490%3663916%195821%195680%
TC: Receiving-an-Order-Line45895100%83735100%45888100%46890100%
Serials-Receiving-Workflow48584100%86692100%47338100%48019100%
Unreceiving-a-Piece8080100%13729100%8013100%8297100%
ULR_TC: Users loan Renewal Transaction31890%48186%29700%31250%


This table contains durations for Check In\Check Out tests. 

RequestsExecutions Response Times (ms)   

#SamplesError %Average90th pct95th pct99th pct
Check-In Controller (MCPT)
CPU=2		24800%4905796661576

Check-Out Controller (MCPT)

CPU=2

24070%1018113912772246

Check-In Controller (QCP1)

CPU=2

22140.63%583659702880

Check-Out Controller (QCP1)

CPU=2

29750.94%953107711661448

Check-In Controller (QCP1)

CPU=0

21910.68%612672719887

Check-Out Controller (QCP1)

CPU=0

29370.89%984110411921442


This table contains durations for Data Import tests. 

 Job ProfileFile

Duration

(h:mm:ss)

Status

PTF Create 2 (MCPT)

CPU=2

25k.mrc

0:15:21COMMITTED

PTF Create 2 (MCPT)

CPU=2

100k.mrc

1:01:25COMMITTED

PTF - Create 2 (QCP1)

CPU=2

25k.mrc

0:14:52COMMITTED

PTF - Create 2 (QCP1)

CPU=2

100k.mrc


0:50:36COMMITTED

PTF - Create 2 (QCP1)

CPU=0

25k.mrc0:14:47COMMITTED

PTF - Create 2 (QCP1)

CPU=0

100k.mrc0:52:34COMMITTED

Comparison

This graph shows the durations of all workflows compared between tests.

Average Case workflows: Part 1.

 Image Added

Average Case workflows: Part 2.


Image Added


High Load workflows: Part 1.

Image Added


High Load workflows: Part 2.

Image Added

High Load workflows: Part 3.

Image Added


Test №1-2-3

The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that nginx-okapi modules used 20% CPU for Test-2 (high load case).

Image Added

Service Memory Utilization

Here we can see that Memory exceed 113% of the unit power mod-data-export-b modules for Test-2 (high load case).

Image Added


Kafka metrics

Image Added


Image Added

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

Image Added

DB Connections

Max number of DB connections was 6000 in maximum fir Test 1, 7500 for Test 2 and 6000 for Test 3.

Image Added

DB load

Image Added                                                                                                                    

Top SQL-queries

Image Added


Test №4

The Baseline MCPT Environment configuration was applied, and CPU=1 was set for all modules.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that CPU did not exceed 95% of the unit CPU power for all of the modules.okapi used 100000% of the CPU power parameter CPU=1 for module.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend..

Image Added


Kafka metrics

Image RemovedImage Added

...

Image Added



DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

Image Added

DB Connections

...

Max number of DB connections was 5568 in maximum.

Resource utilization for Test №2

5600.

Image Added

DB load

 Image Added                                                                                                                    

Top SQL-queries

Image Added



Test №5

The Baseline MCPT Environment configuration  was applied,  and  CPU=0 2 was set for all modules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that okapi used 13% that mod-permissions used 18300% of the absolute CPU power of the container instanceparametr CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added


Kafka metrics

Image Added

Image Added




DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information64% maximum.

Image Added

DB Connections

Max number of DB connections was 57882112.

Image Added

DB load

                          Image Added                                                                                            

Top SQL-queries

Resource utilization for Test №3

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.

Service CPU Utilization

Here we can see that mod-permissions used 20% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 64% maximum.

DB Connections

Max number of DB connections was 2040.

Resource utilization for Test №4

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.

Service CPU Utilization

...

                             

Top SQL-queries

Image Added


Test №6

The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules,  Data Import - Create with 25k and 100k records files tests were run.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that mod-source-record-storage-b used 72000% of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added


Image Added



DB CPU Utilization

DB CPU was 91%99%.

Image Added

DB Connections

Max number of DB connections was 48402500.

Image Added

DB load

Image Added                                                                                                                    

Top SQL-queries

Image Added



Resource utilization for

Test

№5

№7

The Baseline MCPT  QCP1 Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=02 was set for all  services.TTmodules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.

Instance CPU Utilization

Image Added


Service CPU Utilization

Here we can see that okapi used 36% 20% of the absolute CPU power of the container instance.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 42%.

DB Connections

Max number of DB connections was 3650.

DB load

                                                                                                                    

Top SQL-queries

Resource utilization for Test №6

...

. Memory shows stable trend.

Image Added


Kafka metrics

Image Added

Image Added


DB CPU Utilization

DB CPU was 57%.

Image Added

DB Connections

Max number of DB connections was 3650.

Image Added



Test №8

The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all  services.modules,  Data Import - Create with 25k and 100k records files tests were run.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that okapi used 46000% 57000% CPU of unit power.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

...

Image Added

Kafka metrics

Image Added

Image Added


DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

                                                                                                                     

Top SQL-queries

Resource utilization for Test №7

...

Image Added

DB Connections

Max number of DB connections was 971.

Image Added



Test №9

The Baseline QCP1 Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.and CPU=0 was set for all modules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that okapi used 44000% of 15% of the unit absolute CPU power of the container instance.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

...

Image Added


Kafka metrics

Image Added


Image Added



DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

                                                                                                                     

Top SQL-queries

Resource utilization for Test №8

The Baseline MCPT

Image Added

DB Connections

Max number of DB connections was 5150.

Image Added



Test №10

The Baseline QCP1 Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=20 was set for all  services.modules,  Data Import - Create with 25k and 100k records files tests were run.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that okapi used 38% of the unit CPU power.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added


Kafka metrics

Image Added


Image Added



DB CPU Utilization

DB CPU was maximum 53%92%.

Image Added

DB Connections

Max number of DB connections was 3842921.

DB load

                                                                                                                     

Top SQL-queries

Image Added


Appendix

Infrastructure

PTF - Baseline MCPT environment configuration

  • 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.4xlarge

    		128 GiB16 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK tenant
    4 m5.2xlarge ptf-KRaft-mode
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 2version 3.87.0x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...

MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

...

Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database

...

 r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).

  • Test 1: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (average case) MOBIUS test was run.
  • Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (high load case) MOBIUS test was run.
  • Test 3: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules,  Fixed Load (average case) MOBIUS test was run (rerun Test 1).
  • Test 34: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6and CPU=1 was set for all modules,  Fixed Load (average case) MOBIUS test was run.
  • Test 5: The Baseline MCPT Environment configuration was applied, and CPU=02 was set for all servicesmodules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.
  • Test 46:  The The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=02 was set for all servicesmodules,  Data Import - Create with 25k and 100k records files tests were run.
  • Test 57: The Baseline MCPT QCP1 Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=02 was set for all servicesmodules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.
  • Test 68: The  The Baseline MCPT QCP1 Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all servicesmodules,  Data Import - Create with 25k and 100k records files tests were run.
  • Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host"9: The Baseline QCP1 Environment configuration was applied, and CPU=20 was set for all servicesmodules,  Check In\Check Out with 20 users for one tenant on 30 minutes test was run.
  • Test 810: The Baseline MCPT QCP1 Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=20 was set for all servicesmodules,  Data Import - Create with 25k and 100k records files tests were run.