Overview

In this report, PTF investigates the impact of setting CPU allocations to 0 units across all tasks within an AWS ECS cluster. The purpose of this study is to determine whether removing CPU constraints reveals the actual CPU usage of the tasks and to assess how this adjustment affects overall performance. By comparing key workflows across different environments, we aim to identify any potential changes in efficiency, throughput, or resource utilization that may result from setting CPU = 0. The findings from these tests will help inform best practices for resource allocation and performance optimization within ECS clusters.

PERF-959 - Getting issue details... STATUS

Summary

Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.
No memory leaks, memory consumption was stable during all of the tests.
Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.

Test Runs and Results

This table contains durations for all Workflows.

Workflow	Test 1 (average case) CPU=0		Test 2 (high load) CPU=0		Test 3 (average case) CPU=0		Test 4 (average case) CPU=1
	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors	Average (milliseconds)	Errors
AIE_TC: Create Invoices	8350	100%	13458	100%	7421	100%	8251	100%
AIE_TC: Invoices Approve	3180	100%	5251	100%	2960	100%	3088	100%
AIE_TC: Paying Invoices	3289	100%	4919	100%	3060	100%	3285	100%
CICO_TC_Check-In Controller	2323	0%	3322	7%	2228	0%	2313	0%
CICO_TC_Check-Out Controller	4154	0%	5700	10%	3896	0%	3985	0%
CSI_TC:Share local instance	13008	19%	14781	14%	12969	20%	13022	19%
DE_Exporting MARC Bib records custom workflow	52983	0%	379912	95%	78759	0%	100700	0%
DE_Exporting MARC Bib records workflow	42785	0%	402026	98%	73957	0%	109794	0%
EVA_TC: View Account	803	3%	962	3%	700	1%	832	3%
ILR_TC: Create ILR	1527	0%	2215	4%	1451	0%	1518	0%
MSF_TC: mod search by auth query	672	0%	1830	7%	3096	4%	1023	0%
MSF_TC: mod search by boolean query	165	0%	485	2%	737	1%	212	0%
MSF_TC: mod search by contributors	398	0%	1063	3%	1664	2%	604	0%
MSF_TC: mod search by filter query	286	0%	713	2%	1070	2%	417	0%
MSF_TC: mod search by keyword query	284	0%	658	1%	1006	1%	422	0%
MSF_TC: mod search by subject query	407	0%	1112	1%	1530	2%	623	0%
MSF_TC: mod search by title query	1031	0%	2449	1%	2907	1%	1758	0%
OPIH_/oai/records	6042	0%	4649	100%	3587	0%	5448	0%
POO_TC: Add Order Lines	55334	0%	99076	19%	54440	0%	55224	0%
POO_TC: Approve Order	42567	0%	79022	12%	42191	0%	42907	0%
POO_TC Create Order	31933	0%	56468	13%	31406	0%	32166	0%
RTAC_TC: edge-rtac	4150	0%	4726	0%	4084	0%	4099	0%
SDIC_Single Record Import (Create)	13777	19%	20475	13%	13773	20%	13873	19%
SDIU_Single Record Import (Update)	19549	0%	36639	16%	19582	1%	19568	0%
TC: Receiving-an-Order-Line	45895	100%	83735	100%	45888	100%	46890	100%
Serials-Receiving-Workflow	48584	100%	86692	100%	47338	100%	48019	100%
Unreceiving-a-Piece	8080	100%	13729	100%	8013	100%	8297	100%
ULR_TC: Users loan Renewal Transaction	3189	0%	4818	6%	2970	0%	3125	0%

Comparison

This graph shows the durations of all workflows compared between tests.

Average Case workflows: Part 1.

Average Case workflows: Part 2.

High Load workflows: Part 1.

High Load workflows: Part 2.

High Load workflows: Part 3.

Resource utilization for Test №1-2-3

The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules

Service CPU Utilization

Here we can see that nginx-okapi modules used 20for Test-2 (high load case).

Service Memory Utilization

Here we can see that Memory exceed 113% of the unit power mod-data-export-b modules for Test-2 (high load case).

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

DB Connections

Max number of DB connections was 6000 in maximum fir Test 1, 7500 for Test 2 and 6000 for Test 3.

DB load

Top SQL-queries

Resource utilization for Test №2

The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.

Service CPU Utilization

Here we can see that okapi used 13% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

DB Connections

Max number of DB connections was 5788.

DB load

Top SQL-queries

Resource utilization for Test №3

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.

Service CPU Utilization

Here we can see that mod-permissions used 20% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 64% maximum.

DB Connections

Max number of DB connections was 2040.

Resource utilization for Test №4

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.

Service CPU Utilization

Here we can see that okapi used 20% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 91%.

DB Connections

Max number of DB connections was 4840.

DB load

Top SQL-queries

Resource utilization for Test №5

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.TT

Service CPU Utilization

Here we can see that okapi used 36% of the absolute CPU power of the container instance.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 42%.

DB Connections

Max number of DB connections was 3650.

DB load

Top SQL-queries

Resource utilization for Test №6

The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.

Service CPU Utilization

Here we can see that okapi used 46000% CPU of unit power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Inctanse CPU Utilization

Kafka metrics

DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

Top SQL-queries

Resource utilization for Test №7

The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Service CPU Utilization

Here we can see that okapi used 44000% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Inctanse CPU Utilization

Kafka metrics

DB CPU Utilization

DB CPU was 98%.

DB Connections

Max number of DB connections was 5150.

DB load

Top SQL-queries

Resource utilization for Test №8

The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.

Service CPU Utilization

Here we can see that okapi used 38% of the unit CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was maximum 53%.

DB Connections

Max number of DB connections was 3842.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - Baseline MCPT environment configuration

14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs
db.r6g.4xlarge
128 GiB 16 vCPUs
Open Search ptf-test
- Data nodes
  - Instance type - r6g.2xlarge.search
  - Number of nodes - 4
  - Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
  - Instance type - r6g.large.search
  - Number of nodes - 3
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

mcpt modules memory and CPU parameters

Cluster Resources - mcpt-pvt

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
mod-remote-storage	18	mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512
mod-ncip	12	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	14	mod-finance-storage:8.5.0	4	1024	896	1024	700	88	128
mod-agreements	17	mod-agreements:6.0.2	0	1592	1488	128	0	0	0
mod-ebsconet	12	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256
edge-sip2	12	edge-sip2:3.1.1	2	1024	896	128	768	88	128
mod-consortia	18	mod-consortia:1.0.3	2	5136	4776	1024	4416	384	512
mod-organizations	12	mod-organizations:1.8.0	2	1024	896	128	768	88	128
mod-settings	17	mod-settings:1.0.2	2	1024	896	200	768	88	128
edge-dematic	15	edge-dematic:2.1.1	1	1024	896	128	768	88	128
mod-data-import	15	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512
mod-search	18	mod-search:3.0.5	4	2592	2480	2048	1440	512	1024
mod-tags	12	mod-tags:2.1.0	2	1024	896	128	768	88	128
mod-authtoken	22	mod-authtoken:2.14.1	6	1440	1152	1024	922	88	128
edge-courses	12	edge-courses:1.3.0	2	1024	896	128	768	88	128
edge-inventory	6	edge-inventory:1.4.0	2	1024	896	128	768	88	128
mod-inventory-update	12	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128
mod-notify	12	mod-notify:3.1.0	2	1024	896	128	768	88	128
mod-configuration	13	mod-configuration:5.9.2	4	1024	896	128	768	88	128
mod-orders-storage	13	mod-orders-storage:13.6.0	4	1024	896	512	700	88	128
edge-caiasoft	12	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128
mod-login-saml	15	mod-login-saml:2.7.2	2	1024	896	128	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128
mod-password-validator	12	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512
mod-gobi	13	mod-gobi:2.7.1	2	1024	896	128	768	88	128
mod-licenses	15	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512
mod-fqm-manager	12	mod-fqm-manager:1.0.3	2	1024	896	128	768	88	128
edge-dcb	9	edge-dcb:1.0.2	2	1024	896	128	768	88	128
mod-bulk-operations	18	mod-bulk-operations:1.1.9	2	3072	2600	1024	1536	384	512
mod-graphql	12	mod-graphql:1.12.0	0	1024	896	128	768	88	128
mod-finance	12	mod-finance:4.8.0	4	1024	896	128	768	88	128
mod-erm-usage	7	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128
mod-lists	15	mod-lists:1.0.5	2	1024	896	128	768	88	128
mod-copycat	12	mod-copycat:1.5.0	2	1024	896	128	768	88	128
mod-permissions	26	mod-permissions:6.4.0	4	1684	1544	512	1024	384	512
mod-entities-links	17	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024
pub-edge	6	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	13	mod-orders:12.7.1	4	2048	1440	1024	1024	384	512
edge-patron	14	edge-patron:5.0.0.158	2	1024	896	256	768	88	128
edge-ncip	12	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	12	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128
edge-ea-data-export	7	edge-ea-data-export:4.1.0	2	1024	896	128	768	88	128
mod-invoice	13	mod-invoice:5.7.2	4	1440	1152	512	922	88	128
mod-inventory-storage	19	mod-inventory-storage:27.0.4	4	4096	3690	2048	3076	384	512
mod-user-import	12	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	12	mod-sender:1.11.0	2	1024	896	128	768	88	128
edge-oai-pmh	15	edge-oai-pmh:2.7.1	2	1512	1360	1024	1440	384	512
mod-data-export-worker	15	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512
mod-rtac	12	mod-rtac:3.5.0	2	1024	896	128	768	88	128
mod-task-list	7	mod-task-list:1.9.2	2	1024	896	128	768	88	128
mod-circulation-storage	20	mod-circulation-storage:17.1.7	4	2880	2592	1536	1814	384	512
mod-calendar	13	mod-calendar:2.5.0	2	1024	896	128	768	88	128
mod-source-record-storage	21	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512
mod-event-config	12	mod-event-config:2.6.0	2	1024	896	128	768	88	128
mod-courses	12	mod-courses:1.4.8	2	1024	896	128	768	88	128
mod-circulation-item	9	mod-circulation-item:1.0.0	2	1024	896	128	768	88	128
mod-inventory	19	mod-inventory:20.1.6	4	2880	2592	1024	1814	384	512
mod-email	12	mod-email:1.16.0	2	1024	896	128	768	88	128
mod-circulation	20	mod-circulation:24.0.11	4	2880	2592	1536	1814	384	512
mod-pubsub	16	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512
mod-di-converter-storage	17	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128
edge-rtac	15	edge-rtac:2.6.2	2	1024	896	128	768	88	128
edge-orders	15	edge-orders:2.9.1	2	1024	896	128	768	88	128
mod-template-engine	12	mod-template-engine:1.19.1	2	1024	896	128	768	88	128
mod-users	17	mod-users:19.2.2	4	1024	896	128	768	88	128
mod-patron-blocks	14	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128
mod-audit	13	mod-audit:2.8.0	2	1024	896	128	768	88	128
edge-fqm	13	edge-fqm:1.0.1	2	1024	896	128	768	88	128
mod-source-record-manager	18	mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512
nginx-edge	6	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	16	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	1536	0	0	0
okapi-b	8	okapi:5.1.2	3	1684	1440	1536	922	384	512
mod-feesfines	12	mod-feesfines:19.0.0	2	1024	896	128	768	88	128
mod-invoice-storage	13	mod-invoice-storage:5.7.0	4	1872	1536	1024	1024	384	512
edge-users	6	edge-users:1.2.0	2	1024	896	128	768	88	128
mod-dcb	9	mod-dcb:1.0.0	2	1024	896	128	768	88	128
mod-service-interaction	15	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512
mod-data-export	17	mod-data-export:4.8.7	1	1024	896	1024	768	88	128
mod-patron	12	mod-patron:6.0.0	4	1024	896	128	768	88	128
mod-oai-pmh	16	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512
edge-connexion	12	edge-connexion:1.1.1	2	1024	896	128	768	88	128
mod-kb-ebsco-java	12	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	13	mod-notes:5.1.0	2	1024	896	128	952	384	512
mod-data-export-spring	15	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512
mod-login	12	mod-login:7.10.1	2	1440	1298	1024	768	384	512
mod-organizations-storage	13	mod-organizations-storage:4.6.0	4	1024	896	128	768	88	128
pub-okapi	6	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
edge-erm	7	edge-erm:1.0.0	2	1024	896	128	768	88	128
mod-eusage-reports	7	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128

Methodology/Approach

MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.

Baseline MCPT Environment configuration according to tunning environment from previous report: task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).

Test 1: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules, Fixed Load (average case) MOBIUS test was run.
Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules, Fixed Load (high load case) MOBIUS test was run.
Test 3: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules, Fixed Load (average case) MOBIUS test was run (rerun Test 1).
Test 4: The Baseline MCPT Environment configuration was applied, and CPU=1 was set for all modules, Fixed Load (average case) MOBIUS test was run.

Name	Memory GIB	vCPUs
db.r6g.4xlarge	128 GiB	16 vCPUs

PTF - Performance testing of CPU=0 for Services (mcpt)

Overview

Summary

Test Runs and Results

Comparison

Resource utilization for Test №1-2-3

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №2

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №3

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Resource utilization for Test №4

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №5

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №6

Service CPU Utilization

Service Memory Utilization

Inctanse CPU Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №7

Service CPU Utilization

Service Memory Utilization

Inctanse CPU Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Resource utilization for Test №8

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Methodology/Approach