Table of Contents

outline	true

Overview

Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances. We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0. To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-942

Summary

Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.
No memory leaks, memory consumption was stable during all of the tests.
Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.

Test Runs and Results

This table contains durations for all Workflows.

...

Table of Contents

outline	true

Overview

Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances. We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0. To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-942

Summary

Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.
No memory leaks, memory consumption was stable during all of the tests.
Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.

Test Runs and Results

This table contains durations for all Workflows.

Test №1

Introduction: Baseline MCPT Environment configuration.

Objective: The goal of this test is to establish a baseline performance benchmark for the MCPT environment under standard conditions. This test will provide a reference to measure the impact of any subsequent changes, such as modifications in CPU allocation or other resource management strategies. The baseline will be used to assess the performance, responsiveness, and stability of the system with default configurations

Workflow	Test 1 Baseline Env configuration		Test 2 CPU=0		Test 3 x2gd.xlarge CPU=0 instances=6		Test 4 x2gd.xlarge CPU=0 instances=8		Test 5 x2gd.large CPU=0 instances=10		Test 6 r6g.xlarge CPU=2 instances=12(14)		Test 7 r6g.xlarge CPU=2 instances=14 placement strategy (one task per host)		Test 8 x2gd.large CPU=2 instances=14 placement strategy (one task per host)
Workflow	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors
AIE_TC: Create Invoices	7682	100%	8106	100%	50468	100%	15554	100%	92805	100%	10191	100%	9716	100%	91611	100%
AIE_TC: Invoices Approve	2818	100%	3016	100%	24781	100%	7720	100%	51554	100%	4512	100%	4210	100%	51626	100%
AIE_TC: Paying Invoices	2882	100%	3164	100%	32994	100%	9788	100%	64642	100%	5430	100%	4902	100%	65029	100%
CICO_TC_Check-In Controller	2000	0%	2214	0%	19075	0%	6038	0%	32909	0%	3659	0%	3194	0%	30831	0%
CICO_TC_Check-Out Controller	3548	0%	3799	0%	29224	1%	10560	0%	57347	0%	6439	0%	5595	0%	55503	0%
CSI_TC:Share local instance	13050	19%	13073	19%	14265	16%	16192	0%	17361	2%	13782	14%	13339	17%	17460	1%
DE_Exporting MARC Bib records custom workflow	84707	0%	59099	0%	382491	56%	116869	0%	567389	83%	127885	0%	88541	0%	500759	71%
DE_Exporting MARC Bib records workflow	74126	0%	43093	0%	417982	57%	97656	4%	623610	91%	82126	0%	83046	0%	457212	58%
EVA_TC: View Account	519	0%	565	0%	16034	19%	1736	0%	40619	2%	1138	1%	972	1%	37669	1%
ILR_TC: Create ILR	1422	0%	1499	0%	11607	1%	3882	0%	23676	0%	2414	0%	1997	0%	22335	0%
MSF_TC: mod search by auth query	755	2%	668	0%	1888	0%	1500	0%	6180	0%	1080	0%	910	0%	5894	0%
MSF_TC: mod search by boolean query	205	1%	159	0%	594	0%	427	0%	2022	0%	258	0%	238	0%	1871	0%
MSF_TC: mod search by contributors	440	1%	394	0%	856	0%	839	0%	3454	0%	616	0%	530	0%	3300	0%
MSF_TC: mod search by filter query	302	0%	284	0%	512	0%	539	0%	2018	0%	416	0%	362	0%	1924	0%
MSF_TC: mod search by auth keyword query	755310	2%0%	668280	0%	1888521	0%	1500537	0%	61802013	0%	1080416	0%	910361	0%	58941912	0%
MSF_TC: mod search by boolean subject query	205448	1%0%	159406	0%	594797	0%	427778	0%	20223076	0%	258620	0%	238519	0%	18712917	0%
MSF_TC: mod search by contributorstitle query	4401090	1%	3941025	0%	8561435	0%	8391387	0%	34543687	0%	6161432	0%	5301149	0%	33003543	0%	MSF_TC: mod search by filter query
302OPIH_/oai/records	5330	0%	2845404	0%	5129330	0%	5397677	0%	20186881	0%	4163327	0%	3626990	0%	19248335	0%
MSFPOO_TC: mod search by keyword query310Add Order Lines	52142	0%	28054193	0%	521282192	0%	53779206	0%	2013412004	0%	41657735	0%	36157749	0%	1912399917	0%
MSFPOO_TC: mod search by subject query448Approve Order	40656	0%	40642523	0%	797211747	0%	77856446	0%	3076265167	0%	62043930	0%	51943834	0%	2917275935	0%	MSF_TC: mod search by title query	1090
1%	1025POO_TC Create Order	30734	0%	31652	0%	1435107318	0%	138742940	0%	368749234	0%	143232121	0%	114943834	0%	3543175643	0%
OPIH_/oai/records	5330RTAC_TC: edge-rtac	3735	0%	54043828	0%	933016295	0%	76771387	0%	688157195	0%	33274205	0%	69903950	0%	833555595	0%
POO_TC: Add Order Lines	52142	0%	54193	0%	282192	0%	79206	0%	412004	0%	57735	0%	57749	0%	399917	0%
POO_TC: Approve Order	40656	0%	42523	0%	211747	0%	56446	0%	265167	0%	43930	0%	43834	0%	275935	0%
POO_TC Create Order	30734	0%	31652	0%	107318	0%	42940	0%	49234	0%	32121	0%	43834	0%	175643	0%
RTAC_TC: edge-rtac	3735	0%	3828	0%	16295	0%	1387	0%	57195	0%	4205	0%	3950	0%	55595	0%
SDIC_Single Record Import (Create)	13279	19%	13894	19%	45024	16%	17305	0%	79053	2%	14650	14%	14531	17%	74201	1%
SDIU_Single Record Import (Update)	18466	0%	19432	0%	218270	100%	28207	0%	118399	0%	21736	0%	20965	0%	115777	0%
TC: Receiving-an-Order-Line	43765	100%	46104	100%	218270	100%	65242	100%	325230	100%	49024	100%	48538	100%	322267	100%
Serials-Receiving-Workflow	45694	100%	47336	100%	198116	100%	68545	100%	302203	100%	49873	100%	50028	100%	295508	100%
Unreceiving-a-Piece	7823	100%	7757	100%	40059	100%	13335	100%	64155	100%	9018	100%	8717	100%	60295	100%
ULR_TC: Users loan Renewal Transaction	2810	0%	3078	0%	22602	1%	7829	0%	38383	0%	4673	0%	4030	0%	36143	0%

Comparison

This graph shows the total durations of all workflows compared between tests.

Image Removed

SDIC_Single Record Import (Create)	13279	19%	13894	19%	45024	16%	17305	0%	79053	2%	14650	14%	14531	17%	74201	1%
SDIU_Single Record Import (Update)	18466	0%	19432	0%	218270	100%	28207	0%	118399	0%	21736	0%	20965	0%	115777	0%
TC: Receiving-an-Order-Line	43765	100%	46104	100%	218270	100%	65242	100%	325230	100%	49024	100%	48538	100%	322267	100%
Serials-Receiving-Workflow	45694	100%	47336	100%	198116	100%	68545	100%	302203	100%	49873	100%	50028	100%	295508	100%
Unreceiving-a-Piece	7823	100%	7757	100%	40059	100%	13335	100%	64155	100%	9018	100%	8717	100%	60295	100%
ULR_TC: Users loan Renewal Transaction	2810	0%	3078	0%	22602	1%	7829	0%	38383	0%	4673	0%	4030	0%	36143	0%

Comparison

This graph shows the total durations of all workflows compared between tests.

Image Added

Test №1

Introduction: Baseline MCPT Environment configuration.

Objective: The goal of this test is to establish a baseline performance benchmark for the MCPT environment under standard conditions. This test will provide a reference to measure the impact of any subsequent changes, such as modifications in CPU allocation or other resource management strategies. The baseline will be used to assess the performance, responsiveness, and stability of the system with default configurations.

Service CPU Utilization

Here we can see that CPU did not exceed 95% of the unit CPU power for all of the modules.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

Image Added

DB Connections

Max number of DB connections was 5568 in maximum.

Image Added

Test №2

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The goal of this test is to evaluate the performance and behavior of the MCPT environment when CPU reservations are set to 0 for all modules. By not reserving any specific CPU resources, we aim to observe how the system dynamically allocates CPU resources based on availability and demand, and to assess any impacts on module performance, responsiveness, and overall system stability.

Results: Some workflows exhibited significant performance improvements, and the overall average duration for all workflows was better than in Baseline Test №1.

Service CPU Utilization

Here we can see that okapi used 13% of the absolute CPU power of the container instance.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

Image Added

DB Connections

Max number of DB connections was 5788.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test №3

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.

Objective: The objective of this test is to assess the impact of using fewer, more memory-optimized instances (from m6g.2xlarge to x2gd.xlarge) on the performance of the MCPT environment. By reducing the number of instances while selecting a different instance type with a higher memory-to-vCPU ratio, this test aims to observe how the system handles workloads under these conditions and whether the overall efficiency and performance improve.

Results: This configuration led to a significant performance degradation, with performance being four times worse compared to the Baseline Test №1. The reduced number of instances in this setup was clearly insufficient to maintain the required performance levels.

Service CPU Utilization

Here we can see that CPU did not exceed 95% of the unit CPU power for all of the modules.Image Removedthat mod-permissions used 20% of the absolute CPU power of the container instance.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image RemovedImage Added

Kafka metrics

...

Image AddedImage Removed

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving InformationImage Removed64% maximum.

Image Added

DB Connections

Max number of DB connections was 5568 in maximum2040.

Image RemovedImage Added

Test

№2

№4

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all modules services.

Objective: The The goal of this test is to evaluate whether increasing the performance and behavior of the MCPT environment when CPU reservations are set to 0 for all modules. By not reserving any specific CPU resources, we aim to observe how the system dynamically allocates CPU resources based on availability and demand, and to assess any impacts on module performance, responsiveness, and overall system stability.Results: Some workflows exhibited significant performance improvements, and the number of instances from 6 to 8, while maintaining the instance type as x2gd.xlarge and setting CPU=0 for all services, can mitigate the performance degradation observed in Test №3. This test aims to determine the optimal instance count needed to maintain stable performance in the MCPT environment under these conditions

Results: As a result, there was a 3x improvement in overall average duration for all workflows was better than in compared to Test №3. However, performance still lagged by more than 30% when compared to the Baseline Test №1.

Service CPU Utilization

Here we can see that okapi that okapi used 13% 20% of the absolute CPU power of the container instance.

...

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.Image Removed

Image Added

Instance CPU Utilization

Image Added

Kafka metrics

...

Image Added

...

Image Added

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving InformationImage Removed91%.

Image Added

DB Connections

Max number of DB connections was 57884840.

Image RemovedImage Added

DB load

Image Removed Image Added

Top SQL-queries

Image RemovedImage Added

Test

№3

№5

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlargelarge, the number of instances was changed to 610, and CPU=0 was set for all services.

Objective: The objective The goal of this test is to assess evaluate the impact of using fewer, more memory-optimized instances (from m6g.2xlarge to x2gd.xlarge) on the performance of the MCPT environment. By reducing the number of instances while selecting a different instance type with a higher memory-to-vCPU ratio, this test aims to observe how the system handles workloads under these conditions and whether the overall efficiency and performance improve.Results: This configuration led to switching to a smaller instance type, x2gd.large, with a slightly increased instance count of 10, while keeping CPU reservations at 0 for all services. This test aims to find a balance between instance size and count that could potentially optimize performance while reducing costs. We seek to observe how these changes affect the overall system performance compared to previous tests.

Results: The test resulted in a significant performance degradation, with performance being four times worse compared to the Baseline Test №1. The reduced number of instances in this setup was clearly insufficient to maintain the required performance levelsa 7x reduction in performance compared to Baseline Test №1. The change to the x2gd.large instance type and the increased number of instances did not mitigate the performance issues, leading to a much worse overall performance compared to previous configurations in Test №4.

Service CPU Utilization

Here we can see that mod-permissions that okapi used 20% 36% of the absolute CPU power of the container instance.

Image Removed

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Removed

Kafka metrics

Image Removed

DB CPU Utilization

DB CPU was 64% maximum.

Image Removed

DB Connections

Max number of DB connections was 2040.

Image Removed

...

CPU power of the container instance.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Instance CPU Utilization

Image Added

Kafka metrics

Image Added

DB CPU Utilization

DB CPU was 42%.

Image Added

DB Connections

Max number of DB connections was 3650.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test №6

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.

Objective: The goal of this test is to evaluate whether increasing the number of instances from 6 to 8, while maintaining the instance type as x2gd.xlarge and setting CPU=0 for all services, can mitigate the performance degradation observed in Test №3. This test aims to determine the optimal instance count needed to maintain stable performance in the MCPT environment under these conditions

Results: As a result, there was a 3x improvement in overall average duration for all workflows compared to Test №3. However, performance still lagged by more than 30% when compared to the Baseline Test №1changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.

Objective: To assess the performance and behavior of the MCPT environment with a different instance type (r6g.xlarge) and increased CPU allocation (CPU=2 for all services), while using a similar number of instances (12 out of 14). The goal is to determine the impact of these changes on overall system performance, responsiveness, and stability compared to the Baseline configuration.

Results: The test revealed a minor performance degradation of approximately 15-20% in the total duration of all workflows compared to Baseline Test №1. Additionally, we observed uneven distribution of CPU utilization across instances. Some instances experienced high CPU loads of up to 80%, while others had significantly lower utilization, around 5%. This imbalance appears to be due to two main factors: firstly, some modules had only a few tasks concentrated on single instances, and secondly, there were instances with 15 or more tasks, while others had only 0-2 tasks.

Service CPU Utilization

Here we can see that okapi used 20% of the absolute CPU power of the container instance.Image Removed46000% CPU of unit power.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.Image Removed

Image Added

Instance CPU Utilization

Image Added

Kafka metrics

Image RemovedImage Added

DB CPU Utilization

DB CPU was 91%98%.

Image RemovedImage Added

DB Connections

Max number of DB connections was 48405150.

Image RemovedImage Added

DB load

Image Added Image Removed

Top SQL-queries

Image RemovedImage Added

Test

№5

№7

Introduction: The Baseline MCPT Environment configuration was applied, the instance type was changed to

x2gd

r6g.

large

xlarge, the number of instances was changed to

10,

14, placement strategy was updated to "one task per host", and CPU=

0 was set for all services.

Objective: The goal of this test is to evaluate the impact of switching to a smaller instance type, x2gd.large, with a slightly increased instance count of 10, while keeping CPU reservations at 0 for all services. This test aims to find a balance between instance size and count that could potentially optimize performance while reducing costs. We seek to observe how these changes affect the overall system performance compared to previous tests.

Results: The test resulted in a significant performance degradation, with a 7x reduction in performance compared to Baseline Test №1. The change to the x2gd.large instance type and the increased number of instances did not mitigate the performance issues, leading to a much worse overall performance compared to previous configurations in Test №4.

Service CPU Utilization

Here we can see that okapi used 36% of the absolute CPU power of the container instance.

Image Removed

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Removed

Kafka metrics

Image Removed

DB CPU Utilization

DB CPU was 42%.

Image Removed

DB Connections

Max number of DB connections was 3650.

Image Removed

DB load

Image Removed

Top SQL-queries

Image Removed

Resource utilization for Test №6

The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.

Introduction:

Objective:

Results:

2 was set for all services.

Objective: The objective of this test is to evaluate the performance and behavior of the MCPT environment with a consistent instance type (r6g.xlarge), a fixed number of instances (14), and a new placement strategy ("one task per host"). By setting CPU=2 for all services, we aim to assess the impact of these changes on overall system performance, responsiveness, and stability, and to address the uneven CPU utilization observed in Test №6.

Results: The test revealed a minor performance improvement of approximately 3%-5% compared to Test №6. However, the total duration of all workflows showed a performance degradation of around 10%-15% compared to Baseline Test №1. The updated placement strategy of "one task per host" had a positive effect on addressing the previously observed imbalance but did not fully achieve the expected results. We still observed instances with high CPU loads (up to 80%) and others with significantly lower utilization (around 5%). Despite the more balanced distribution of tasks, some instances continued to have a high number of tasks (15+), while others had very few (2), leading to persistent CPU load imbalance.

Service CPU Utilization

Here we can see that okapi used

...

44000% of the unit CPU power.

...

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

...

Image Added

...

Instance CPU Utilization

...

Image Added

Kafka metrics

...

Image Added

...

Image Added

DB CPU Utilization

DB CPU was 98%.

...

Image Added

DB Connections

Max number of DB connections was 5150.

...

Image Added

DB load

...

Image Added

Top SQL-queries

...

Image Added

Resource utilization for

Test

№7

№8

Introduction:The

Baseline MCPT Environment configuration

was applied, the

instance type

was changed to

r6g

x2gd.

xlarge

large, the

number of instances

was changed to

14, placement strategy was updated to

"one task per host",

and

CPU=2

was set for all

services.

Introduction:

Objective:

Results:

Service CPU Utilization

Here we can see that okapi used 44000% of the unit CPU power.

Image Removed

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Removed

Inctanse CPU Utilization

Image Removed

Kafka metrics

Image Removed

DB CPU Utilization

DB CPU was 98%.

Image Removed

DB Connections

Max number of DB connections was 5150.

Image Removed

DB load

Image Removed

Top SQL-queries

Image Removed

Resource utilization for Test №8

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Introduction:

Objective:

Results:

The objective of this test is to evaluate the performance and behavior of the MCPT environment using the x2gd.large instance type with a consistent number of instances (14) and the "one task per host" placement strategy. By setting CPU=2 for all services, we aim to assess the impact of these configurations on overall system performance, responsiveness, and stability, and to compare the results with previous tests to determine the effectiveness of the new setup.

Results: The test revealed a significant performance degradation, with the total average duration for all workflows being approximately 5 times worse compared to the Baseline Test №1. Additionally, there was an imbalance in CPU utilization across instances, indicating that some instances were underutilized while others were overloaded.

Service CPU Utilization

Here we can see that okapi used 38% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Instance CPU Utilization

Image Added

Kafka metrics

DB CPU Utilization

DB CPU was maximum 53%.

DB Connections

Max number of DB connections was 3842.

DB load

Top SQL-queries

...

PTF - Baseline MCPT environment configuration

14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer

Name Memory GIB vCPUs
db.r6g.4xlarge
128 GiB 16 vCPUs
Open Search ptf-test
- Data nodes
  - Instance type - r6g.2xlarge.search
  - Number of nodes - 4
  - Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
  - Instance type - r6g.large.search
  - Number of nodes - 3
- MSK
tenant
- 4 m5.2xlarge ptf-KRaft-mode
  - 2 brokers, kafka.m7g.xlarge brokers in 2 zones
  - Apache Kafka
  version 2
  - version 3.
  8
  - 7.
  0
  - x
  - EBS storage volume per broker 300 GiB
  - auto.create.topics.enable=true
  - log.retention.minutes=480
  - default.replication.factor=3

Expand

title	mcpt modules memory and CPU parameters

Cluster Resources - mcpt-pvt

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
mod-remote-storage	18	mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512
mod-ncip	12	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	14	mod-finance-storage:8.5.0	4	1024	896	1024	700	88	128
mod-agreements	17	mod-agreements:6.0.2	0	1592	1488	128	0	0	0
mod-ebsconet	12	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256
edge-sip2	12	edge-sip2:3.1.1	2	1024	896	128	768	88	128
mod-consortia	18	mod-consortia:1.0.3	2	5136	4776	1024	4416	384	512
mod-organizations	12	mod-organizations:1.8.0	2	1024	896	128	768	88	128
mod-settings	17	mod-settings:1.0.2	2	1024	896	200	768	88	128
edge-dematic	15	edge-dematic:2.1.1	1	1024	896	128	768	88	128
mod-data-import	15	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512
mod-search	18	mod-search:3.0.5	4	2592	2480	2048	1440	512	1024
mod-tags	12	mod-tags:2.1.0	2	1024	896	128	768	88	128
mod-authtoken	22	mod-authtoken:2.14.1	6	1440	1152	1024	922	88	128
edge-courses	12	edge-courses:1.3.0	2	1024	896	128	768	88	128
edge-inventory	6	edge-inventory:1.4.0	2	1024	896	128	768	88	128
mod-inventory-update	12	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128
mod-notify	12	mod-notify:3.1.0	2	1024	896	128	768	88	128
mod-configuration	13	mod-configuration:5.9.2	4	1024	896	128	768	88	128
mod-orders-storage	13	mod-orders-storage:13.6.0	4	1024	896	512	700	88	128
edge-caiasoft	12	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128
mod-login-saml	15	mod-login-saml:2.7.2	2	1024	896	128	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128
mod-password-validator	12	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512
mod-gobi	13	mod-gobi:2.7.1	2	1024	896	128	768	88	128
mod-licenses	15	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512
mod-fqm-manager	12	mod-fqm-manager:1.0.3	2	1024	896	128	768	88	128
edge-dcb	9	edge-dcb:1.0.2	2	1024	896	128	768	88	128
mod-bulk-operations	18	mod-bulk-operations:1.1.9	2	3072	2600	1024	1536	384	512
mod-graphql	12	mod-graphql:1.12.0	0	1024	896	128	768	88	128
mod-finance	12	mod-finance:4.8.0	4	1024	896	128	768	88	128
mod-erm-usage	7	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128
mod-lists	15	mod-lists:1.0.5	2	1024	896	128	768	88	128
mod-copycat	12	mod-copycat:1.5.0	2	1024	896	128	768	88	128
mod-permissions	26	mod-permissions:6.4.0	4	1684	1544	512	1024	384	512
mod-entities-links	17	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024
pub-edge	6	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	13	mod-orders:12.7.1	4	2048	1440	1024	1024	384	512
edge-patron	14	edge-patron:5.0.0.158	2	1024	896	256	768	88	128
edge-ncip	12	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	12	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128
edge-ea-data-export	7	edge-ea-data-export:4.1.0	2	1024	896	128	768	88	128
mod-invoice	13	mod-invoice:5.7.2	4	1440	1152	512	922	88	128
mod-inventory-storage	19	mod-inventory-storage:27.0.4	4	4096	3690	2048	3076	384	512
mod-user-import	12	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	12	mod-sender:1.11.0	2	1024	896	128	768	88	128
edge-oai-pmh	15	edge-oai-pmh:2.7.1	2	1512	1360	1024	1440	384	512
mod-data-export-worker	15	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512
mod-rtac	12	mod-rtac:3.5.0	2	1024	896	128	768	88	128
mod-task-list	7	mod-task-list:1.9.2	2	1024	896	128	768	88	128
mod-circulation-storage	20	mod-circulation-storage:17.1.7	4	2880	2592	1536	1814	384	512
mod-calendar	13	mod-calendar:2.5.0	2	1024	896	128	768	88	128
mod-source-record-storage	21	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512
mod-event-config	12	mod-event-config:2.6.0	2	1024	896	128	768	88	128
mod-courses	12	mod-courses:1.4.8	2	1024	896	128	768	88	128
mod-circulation-item	9	mod-circulation-item:1.0.0	2	1024	896	128	768	88	128
mod-inventory	19	mod-inventory:20.1.6	4	2880	2592	1024	1814	384	512
mod-email	12	mod-email:1.16.0	2	1024	896	128	768	88	128
mod-circulation	20	mod-circulation:24.0.11	4	2880	2592	1536	1814	384	512
mod-pubsub	16	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512
mod-di-converter-storage	17	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128
edge-rtac	15	edge-rtac:2.6.2	2	1024	896	128	768	88	128
edge-orders	15	edge-orders:2.9.1	2	1024	896	128	768	88	128
mod-template-engine	12	mod-template-engine:1.19.1	2	1024	896	128	768	88	128
mod-users	17	mod-users:19.2.2	4	1024	896	128	768	88	128
mod-patron-blocks	14	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128
mod-audit	13	mod-audit:2.8.0	2	1024	896	128	768	88	128
edge-fqm	13	edge-fqm:1.0.1	2	1024	896	128	768	88	128
mod-source-record-manager	18	mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512
nginx-edge	6	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	16	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	1536	0	0	0
okapi-b	8	okapi:5.1.2	3	1684	1440	1536	922	384	512
mod-feesfines	12	mod-feesfines:19.0.0	2	1024	896	128	768	88	128
mod-invoice-storage	13	mod-invoice-storage:5.7.0	4	1872	1536	1024	1024	384	512
edge-users	6	edge-users:1.2.0	2	1024	896	128	768	88	128
mod-dcb	9	mod-dcb:1.0.0	2	1024	896	128	768	88	128
mod-service-interaction	15	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512
mod-data-export	17	mod-data-export:4.8.7	1	1024	896	1024	768	88	128
mod-patron	12	mod-patron:6.0.0	4	1024	896	128	768	88	128
mod-oai-pmh	16	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512
edge-connexion	12	edge-connexion:1.1.1	2	1024	896	128	768	88	128
mod-kb-ebsco-java	12	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	13	mod-notes:5.1.0	2	1024	896	128	952	384	512
mod-data-export-spring	15	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512
mod-login	12	mod-login:7.10.1	2	1440	1298	1024	768	384	512
mod-organizations-storage	13	mod-organizations-storage:4.6.0	4	1024	896	128	768	88	128
pub-okapi	6	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
edge-erm	7	edge-erm:1.0.0	2	1024	896	128	768	88	128
mod-eusage-reports	7	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128

...

Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report: task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).
Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Name	Memory GIB	vCPUs
db.r6g.4xlarge	128 GiB	16 vCPUs

Page Comparison

Versions Compared

Old Version 36

New Version Current

Key

Overview

Summary

Test Runs and Results

Overview

Summary

Test Runs and Results

Test №1

Comparison

Comparison

Test №1

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Test №2

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №3

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Test

№4

Service CPU Utilization

Service Memory Utilization

Instance CPU Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test

№5

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Service Memory Utilization

Instance CPU Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №6

Service CPU Utilization

Service Memory Utilization

Instance CPU Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test

№7

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №6

Service CPU Utilization

Service Memory Utilization

Instance CPU Utilization