Table of Contents

outline	true

Overview

The primary objective of testing was to evaluate the performance of the Baseline MCPT Environment configuration while attempting to optimize costs by adjusting instance types and reducing the number of instances. The tests were designed to compare the performance outcomes across different configurations, including variations in instance types and counts within multiple Auto Scaling Groups (ASGs). By systematically modifying these variables, the goal was to maintain or improve the performance observed in the baseline configuration while achieving cost efficiency.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-962

Summary

Through a series of experiments involving different placement strategies, instance types, and total instance counts, we found that the performance remained consistent when using these two configuration:
- six c7g.large instances (three dedicated to the okapi service and three allocated to mod-courses, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, and mod-data-import services) alongside ten r7g.xlarge instances for all other services, with the CPU parameter set to 2 for all services.
- three c7g.large instances dedicated to the okapi service alongside five r7g.2xlarge instances for all other services, with the CPU parameter set to 2 for all services.
Notably, both environment configuration offers a 40% cost reduction compared to the existing setup, making it a more economical option without compromising on performance.
Configurations with three c7g.large instances for the okapi service and five r7g.2xlarge instances for all other services show the best performance across all experiments. So it will be more tests in this ticket.
Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.

Test Runs

...

Test Results

This table contains durations for all Workflows.

...

Test 2

(Repeat 1)

...

Test 4

(Repeat 3)

...

Test 6

(Repeat 5)

...

Comparison

This graph shows the durations of all workflows compared between the best test results.

Average Case workflows: Part 1.

Image Removed

Average Case workflows: Part 2.

Image Removed

Test №1

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of test was to evaluate the performance of the MCPT environment by applying the baseline configuration, while adjusting the CPU allocation for all services to 2 vCPUs. Additionally, the test aimed to assess the impact of using two distinct autoscaling groups with different instance types: c7g.large for the Okapi service and r7g.xlarge for all other services. By running the Fixed Load (average case) MOBIUS test, the goal was to determine if this modified configuration could achieve similar or improved performance compared to the baseline, while potentially optimizing resource allocation and cost.

Results: Almost all workflows shows significant performance degradation.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi modules used 42k% CPU power parameter CPU=2 for module.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information.

DB Connections

Max number of DB connections was 5600 in maximum.

Test №2

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others modules. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to maintain the use of m6g.2xlarge instances for the main modules, as in the Baseline configuration, while introducing a second autoscaling group with c7g.large instances specifically for the Okapi service. This setup aims to explore the performance and cost-effectiveness of separating the Okapi service onto a smaller instance type while keeping the main services on the larger m6g.2xlarge instances.

Results: We observed nearly identical performance results in over half of the workflows compared to the Baseline configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 47k% of the CPU power parameter CPU=2 for module.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

DB Connections

Max number of DB connections was 5710.

Test №3

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others services. Fixed Load (average case) MOBIUS test was run (Repeat Test 2).

Objective: The objective of this test was to validate the consistency of performance observed in Test 2 by repeating the same configuration.

Results: We observed nearly identical performance results almost for all the workflows compared to the Baseline configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of the parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99% maximum.

DB Connections

Max number of DB connections was 5500.

Test №4

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to add more services with lower activity levels to the first autoscaling group, increasing the number of c7g.large instances from 3 to 6. For the main services, r7g.xlarge instances were used with a total of 10 instances, which are more cost-effective compared to the Baseline configuration with r6g.4xlarge.

Results: We observed a better performance compared to previous test.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5810.

DB load

Top SQL-queries

Test №5

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 4).

Objective: The objective of this test was to validate the consistency of performance observed in Test 4 by repeating the same configuration.

Results: We confirmed performance improvements with this configuration, also a several workflows shoes better performance than in Test 4.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5500.

DB load

Top SQL-queries

Test №6

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run.

Objective: The goal of this test was to exclude several unstable workflows and observe whether their removal would have a noticeable impact on the performance of the remaining workflows. Specifically, by omitting the DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow, the test aimed to assess if the stability and performance of the other workflows would improve under the same configuration.

Results: We observed minor performance improvements across almost all workflows. However, these improvements were not substantial. It appears that the observed effects are relatively small and can be attributed to a reduced overall load in the environment.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

Table of Contents

outline	true

Overview

The primary objective of testing was to evaluate the performance of the Baseline MCPT Environment configuration while attempting to optimize costs by adjusting instance types and reducing the number of instances. The tests were designed to compare the performance outcomes across different configurations, including variations in instance types and counts within multiple Auto Scaling Groups (ASGs). By systematically modifying these variables, the goal was to maintain or improve the performance observed in the baseline configuration while achieving cost efficiency.

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-962

Summary

Through a series of experiments involving different placement strategies, instance types, and total instance counts, we found that the performance remained consistent when using these configurations:
- three c7g.large instances dedicated to the okapi service alongside five r7g.2xlarge instances for all other services, with the CPU parameter set to 2 for all services.
- five r7g.2xlarge instances for all services, with the CPU parameter set to 2 for all services.
Optimized environment configurations offers a 20-40% cost reduction compared to the existing setup, making it a more economical option without compromising on performance.
Configurations with three c7g.large instances for the okapi service and five r7g.2xlarge instances for all other services show the best performance across all experiments.
In fact, some workflows show better performance with this new setup than correct infrastructures.
The CPU utilization on EC2 level better now - around 30-60%, previously it was under 20%.

AWS Configuration Costs

Cluster	Instance Type	Cost per Month (USD)	Number of Instances	Total Cost per Cluster (USD)
QCP1	m6g.2xlarge	$221.76	10	$2,217.60
MCPT	m6g.2xlarge	$221.76	14	$3,104.64
Optimized Infrastructure Two Auto Scaling Groups	c7g.large	$52.20	3	$1,698.84
Optimized Infrastructure Two Auto Scaling Groups	r7g.2xlarge	$308.45	5	$1,698.84
Optimized Infrastructure One Auto Scaling Groups	r7g.2xlarge	$308.45	5	$1,542.25

Cost Comparison (Before vs After)

Cluster	Previous Total Cost (USD)	New Total Cost (USD)	Percentage Saving (%)
QCP1	$2,217.60	$1,698.84	23.39%
MCPT	$3,104.64	$1,698.84	45.28%

Test Runs

Test #	Description	Status
Test 1	Instance type: m6g.2xlarge. Instances count: 10.	Completed
Test 2	Instance type: m6g.2xlarge. Instances count: 10 (Repeat Test 1).	Completed
Test 3	Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services.	Completed
Test 4	Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services (Repeat Test 3).	Completed
Test 5	CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules.	Completed
Test 6	CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules (Repeat Test 5).	Completed
Test 7	CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules.	Completed
Test 8	CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for all services.
Test 9	CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 8).
Test 10	CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services.
Test 11	CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 10).
Test 12	CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 11).

Test Results

This table contains durations for all Workflows.

Workflows	Test 1		Test 2 (Repeat 1)		Test 3		Test 4 (Repeat 3)		Test 5		Test 6 (Repeat 5)		Test 7		Test 8		Test 9 (Repeat 8)		Test 10		Test 11 (Repeat 10)		Test 12 (Repeat 11)
	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors	Average response time (milliseconds)	Errors
DATA IMPORT	0:52:03		0:44:55		0:46:07		0:47:09		0:51:41		0:58:35		1:00:03		0:43:53		0:47:06		0:55:30		0:45:46		0:44:09
DATA EXPORT	0:58:11		0:44:43		0:47:41		0:50:32		0:38:59		0:45:53		0:48:26	not finished for main	0:45:41		0:48:49		0:56:49		0:42:16		0:44:26
CICO_TC_Check-In Controller	1163	0%	948	0%	932	0%	958	0%	849	0%	895	0%	940	0%	912	0%	993	0%	1176	0%	892	0%	967	0%
CICO_TC_Check-Out Controller	1697	0%	1481	0%	1408	0%	1428	0%	1318	0%	1318	0%	1367	0%	1345	0%	1445	0%	1675	0%	1371	0%	1467	0%
DE_Exporting MARC Bib records workflow	2528	0%	3818	0%	3675	0%	2830	0%	1918	0%	2223	0%	1865	0%	3363	0%	2420	0%	1872	0%	3398	0%	5033	0%
ILR_TC: Create ILR	1023	0%	874	0%	730	0%	804	0%	624	0%	662	0%	802	0%	820	0%	767	0%	1116	0%	877	0%	780	0%
ILR_TC: Get ItemId	132	0%	131	0%	107	0%	114	0%	112	0%	109	0%	109	0%	116	0%	127	0%	165	0%	136	0%	130	0%
MSF_TC: mod search by auth query	4830	0%	4835	6%	5480	6%	7658	10%	7570	19%	2175	0%	2474	0%	6094	10%	4119	12%	2938	0%	5546	11%	7075	18%
MSF_TC: mod search by boolean query	469	0%	621	3%	1456	2%	1027	5%	1876	5%	256	1%	605	0%	1343	5%	527	4%	333	0%	1197	7%	1263	6%
MSF_TC: mod search by contributors	475	0%	1655	3%	1805	6%	1714	3%	3467	8%	496	0%	503	0%	1966	4%	755	6%	465	0%	2090	5%	2036	8%
MSF_TC: mod search by filter query	229	0%	844	2%	703	2%	1137	0%	1573	2%	217	0%	262	0%	1100	2%	429	3%	238	0%	944	4%	1068	6%
MSF_TC: mod search by keyword query	228	0%	519	3%	1161	4%	1245	8%	1312	4%	198	0%	256	0%	1144	3%	434	2%	250	0%	979	4%	1018	8%
MSF_TC: mod search by subject query	577	0%	1846	3%	1825	3%	1634	7%	2328	2%	445	0%	539	0%	2163	3%	840	7%	473	0%	1476	6%	1919	8%
MSF_TC: mod search by title query	2141	0%	3251	3%	3770	1%	4598	2%	4887	8%	2038	0%	2294	0%	4285	5%	3214	6%	2054	0%	4145	7%	4437	9%
DI_TC: Importing MARC records workflow Transaction &{tenant}	1059451	0%	20423	0%	935626	0%	950403	0%	1047836	0%	17946	0%	1210429	33%	17258	0%	955349	0%	2238208	0%	18974	0%	18762	0%
PRV_TC: View Patron record Group	347	0%	310	0%	248	0%	238	0%	213	0%	284	0%	313	0%	265	0%	254	0%	355	0%	242	0%	273	0%
ULR_TC: Users loan Renewal Transaction	1059451	0%	1852	0%	1528	0%	1717	0%	1551	0%	1595	0%	1749	0%	1526	0%	1761	0%	4117	0%	1496	0%	1617	0%

Comparison

This graph shows the durations of all workflows compared between the best test results.

Part 1

Image Added

Part 2

Image Added

Part 3

Image Added

Test №1

Introduction: Test 1: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run.

Objective: The objective of test was to evaluate the performance of the MCPT environment by applying the baseline configuration.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that mod-inventory-b modules used 73% CPU.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was 99%97% average.

Image Added

DB Connections

Max number of DB connections was 60001250 in maximum.

Image Added

DB load

Top SQL-queries

Test №7 - 8

Introduction:

Test 7: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 6).
Test 8: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.

...

Image Added

Top SQL-queries

Image Added

Test №2

Introduction: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run (Repeat Test 1).

Objective:The objective of this test was to validate the consistency of performance observed in Test

5 and Test 6. This was achieved

1 by repeating the same configuration.

Results:

We see performance improvements for Test 8

Results was almost the same for all workflows.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-inventory-b modules used 102% CPU in maximum.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

95% average

Image Added

DB Connections

Max number of DB connections was

...

1250.

DB load

Top SQL-queries

Test №9 - 10

Introduction:

Test 9: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 7 after Terminate Instances r7g.xlarge).
Test 10: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 8 after Terminate Instances r7g.xlarge).

Objective: The objective of these tests was to validate the consistency of performance observed in Test 7 and Test 8. This was achieved by repeating the same configuration and applying new random task locations per instance after terminating the r7g.xlarge instances.

Results: Performance result were worse on 50% for several workflows after terminating the r7g.xlarge instances and applying new random task locations per instance.

Instance CPU Utilization

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test №3-4

Introduction: Test 3: The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run. Test 3: Repeat Test 1.

Objective: The objective of test was to evaluate the performance of the QCP1 environment by applying two distinct autoscaling groups with different instance types: c7g.large for the Okapi service and r7g.xlarge for all other services. By running test, the goal was to determine if this modified configuration could achieve similar or improved performance compared to the baseline, while potentially optimizing resource allocation and cost.

Results: We observed nearly identical performance results almost for all the workflows compared to the Baseline configuration and repeated it in Test №4.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-inventory-b used 78k% of the

...

CPU

...

.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

97% maximum.

Image Added

DB Connections

Max number of DB connections was

...

2200.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test

№11

№5-6-7

Introduction: The

Baseline

MCPT

QCP1 Environment configuration

was applied,

and

CPU=2

was set for all

services

modules, used two autoscaling groups, 1st with

3 Instance Type: c7g.large

for

okapi service and

11 Instance

5 Instance Type:

r7g.xlarge for others services. Fixed Load (average case) MOBIUS test

c7g.large for others services, qcp1_MiniMaster.jmx script was run.

Objective:

The goal of this test was to replace three c7g.large instances with one additional r7g.xlarge instance for all services, while retaining three c7g.large instances specifically for the okapi service.Results: The performance was worse compared to when we used the additional three c7g.large instances for services with smaller workloads

Repeat tests to confirm performance.

Results: Performance was confirmed at the same like baseline.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-data-export-b used 82k% of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

97%.

Image Added

DB Connections

Max number of DB connections was

...

DB load

1250.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test

№12

№8-

13

9

Introduction: The Baseline

MCPT

QCP1 Environment configuration was applied, and CPU=2 was set for all

services

modules, used

two autoscaling groups, 1st with 3

ONE autoscaling group with 5 Instance Type: c7g.large for

okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test

all services, qcp1_MiniMaster.jmx script was run.

Objective:

The objective

The goal of

this

test

was to validate the degradation of performance observed in Test 11 by repeating the same configuration

is to put okapi service in one tier with all services and compare performance with previous result and baseline result.

Results:

We confirmed performance degradation with this configuration

Performance was confirmed at the same like baseline.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

94%.

Image Added

DB Connections

Max number of DB connections was

...

1265.

...

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test

№14

№10

Introduction: The Baseline

MCPT

QCP1 Environment configuration was applied, and CPU=2 was set for all

services

modules except CPU=2048 for mod-search, used

two autoscaling groups, 1st with 3

ONE autoscaling group with 5 Instance Type: c7g.large for

okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same

all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results:

We confirmed performance degradation with this configuration

Performance was confirmed at the same like previous one.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-data-export-b used 116k% MAX of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

94%.

Image Added

DB Connections

Max number of DB connections was

...

1220.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test

№15

№11

Introduction: The Baseline

MCPT

QCP1 Environment configuration was applied, and CPU=2 was set for all

services

modules except CPU=2048 for mod-search, used

two autoscaling groups, 1st with 3

ONE autoscaling group with 5 Instance Type: c7g.large for

okapi and mod-authtoken service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective:

Results: We confirmed performance degradation with this configuration

all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results: Performance was confirmed at the same like previous one.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory

...

Kafka metrics

DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5150.

...

leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was 94%.

Image Added

DB Connections

Max number of DB connections was 1200.

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Test

№16 - 17

№12

Introduction:

Test 16: The Baseline MCPT

The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all

services

modules except CPU=2048 for mod-search, used

two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: r7g.2xlarge for others services. Fixed Load (average case) MOBIUS test was run. Test 17: Repeat Test 16 with the same environment configuration.

Objective: The objective of this test was to reduce the number of instances in the main group from 10 to 5, while using higher capacity r7g.2xlarge instances instead of r7g.xlarge

Results: We observed performance improvements in Test 16 compared to Test 8, and by repeating the same configuration in Test 17, we confirmed these performance improvements

ONE autoscaling group with 5 Instance Type: c7g.large for all services, qcp1_MiniMaster.jmx script was run

Objective: The goal of test is confirm the test result with previous configuration.

Results: Performance was confirmed at the same like previous one.

Instance CPU Utilization

Image Added

Service CPU Utilization

Here we can see that

...

mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.

Image Added

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Image Added

Kafka metrics

Image Added

OpenSearch metrics

Image Added

DB CPU Utilization

DB CPU was

...

94%.

Image Added

DB Connections

Max number of DB connections was

...

1265.

...

Image Added

DB load

Image Added

Top SQL-queries

Image Added

Appendix

Infrastructure

PTF - Baseline QCP1 environment configurationenvironment configuration (was changed during testing)

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer

Name Memory GIB vCPUs
db.r6g.xlarge
32 GB 4 vCPUs
Open Search ptf-test
- Data nodes
  - Instance type - r6g.2xlarge.search
  - Number of nodes - 4
  - Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
  - Instance type - r6g.large.search
  - Number of nodes - 3
MSK fse-tenant
- 2 brokers, kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Expand

title	qcp1 modules memory and CPU parameters

Cluster Resources - qcp1-pvt (Thu Sep 12 09:30:08 UTC 2024)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	6	mod-remote-storage:3.2.0	2	4920	4472	0	3960	512	512
mod-ncip	6	mod-ncip:1.14.4	2	1024	896	0	768	88	128
mod-finance-storage	6	mod-finance-storage:8.6.0	2	1024	896	0	700	88	128
mod-agreements	6	mod-agreements:7.0.0	2	1592	1488	0	0	0	0
mod-ebsconet	6	mod-ebsconet:2.2.0	2	1248	1024	0	700	128	256
mod-organizations	6	mod-organizations:1.9.0	2	1024	896	0	768	88	128
mod-consortia	4	mod-consortia:1.1.0	2	5136	4776	0	4416	384	512
edge-sip2	4	edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	0	768	88	128
mod-serials-management	6	mod-serials-management:1.0.0	2	2480	2312	0	1792	384	512
mod-settings	6	mod-settings:1.0.3	2	1024	896	0	768	88	128
mod-data-import	9	mod-data-import:3.1.0	1	2048	1844	0	1292	384	512
edge-dematic	6	edge-dematic:2.2.0	1	1024	896	0	768	88	128
mod-search	6	mod-search:3.2.0	2	2592	2480	0	1440	512	1024
mod-inn-reach	4	mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	0	2880	512	1024
mod-tags	6	mod-tags:2.2.0	2	1024	896	0	768	88	128
edge-courses	6	edge-courses:1.4.0	2	1024	896	0	768	88	128
mod-authtoken	10	mod-authtoken:2.15.1	2	1440	1152	0	922	88	128
mod-inventory-update	6	mod-inventory-update:3.3.0	2	1024	896	0	768	88	128
mod-notify	6	mod-notify:3.2.0	2	1024	896	0	768	88	128
mod-configuration	6	mod-configuration:5.10.0	2	1024	896	0	768	88	128
mod-orders-storage	6	mod-orders-storage:13.7.0	2	1024	896	0	700	88	128
edge-caiasoft	6	edge-caiasoft:2.2.0	2	1024	896	0	768	88	128
mod-login-saml	6	mod-login-saml:2.8.0	2	1024	896	0	768	88	128
mod-erm-usage-harvester	6	mod-erm-usage-harvester:4.5.0	2	1024	896	0	768	88	128
mod-licenses	6	mod-licenses:6.0.0	2	2480	2312	0	1792	384	512
mod-gobi	6	mod-gobi:2.8.0	2	1024	896	0	768	88	128
mod-password-validator	6	mod-password-validator:3.2.0	2	1440	1298	0	768	384	512
mod-bulk-operations	6	mod-bulk-operations:2.0.0	2	3072	2600	0	1536	384	512
mod-fqm-manager	6	mod-fqm-manager:2.0.1	2	1024	896	0	768	88	128
edge-dcb	6	edge-dcb:1.1.0	2	1024	896	0	768	88	128
mod-graphql	6	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	6	mod-finance:4.9.0	2	1024	896	0	768	88	128
mod-erm-usage	6	mod-erm-usage:4.7.0	2	2800	2550	0	1800	384	512
mod-batch-print	7	mod-batch-print:1.1.0	2	1024	896	0	768	88	128
mod-copycat	6	mod-copycat:1.6.0	2	1024	896	0	768	88	128
mod-lists	6	mod-lists:2.0.0	2	1024	896	0	768	88	128
mod-entities-links	7	mod-entities-links:3.0.0	2	2592	2480	0	1440	0	1024
mod-permissions	11	mod-permissions:6.5.0	2	1684	1544	0	1024	384	512
pub-edge	4	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	6	mod-orders:12.8.0	2	2048	1740	0	1024	384	512
edge-patron	3	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	46	pubedge-edgencip:20231.069.142	2	1024	896	0	768	0880	128
mod-ordersmarc-migrations	62	mod-marc-ordersmigrations:121.80.0-SNAPSHOT.5	20	2048	17401764	010241024	1536	38488	512128
edge-patroninn-reach	34	edge-inn-patronreach:53.1.1-SNAPSHOT.045	2	1024	896	2560	768	88	128
edgemod-users-ncip	6	edge-ncip:1.9.2bl	6	mod-users-bl:7.7.0	2	1440	1152	0	922	88	128
mod-oa	5	mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	0	768	88	128256
mod-marcinventory-migrationsstorage	23	mod-marcinventory-migrationsstorage:27.1.0.0-SNAPSHOT.50	2	20484096	17643690	10242048	15363076	88384	128512
edgemod-inn-reachinvoice	48	edgemod-inn-reachinvoice:35.18.1-SNAPSHOT.45	2	10241440	8961152	0	768922	88	128
mod-usersuser-blimport	6	mod-usersuser-blimport:73.78.0	2	14401024	1152896	0	922768	88	128
mod-oasender	57	mod-oasender:2.1.0-SNAPSHOT12.620	2	1024	896	0	768	88	256128
modedge-inventoryoai-storagepmh	36	modedge-inventoryoai-storagepmh:272.19.0	2	40961512	36901360	20480	30761440	384	512
mod-invoice8data-export-worker	6	mod-data-export-invoiceworker:53.82.1	2	14403072	11522800	0	9222048	88384	128512
mod-user-importrtac	63	mod-user-importrtac:3.86.0	2	1024	896	0128	768	88	128
mod-sendercirculation-storage	73	mod-circulation-senderstorage:117.122.0	2	10242880	8962592	01536	7681814	88384	128512
edgemod-oai-pmhcalendar	6	edgemod-oai-pmhcalendar:23.91.0	2	15121024	1360896	0	1440768	38488	512128
mod-datasource-exportrecord-workerstorage	618	mod-datasource-exportrecord-workerstorage:35.28.17	2	30725600	28005000	020482048	3500	384	512
mod-event-rtacconfig	36	mod-event-rtacconfig:32.67.0	2	1024	896	1280	768	88	128
mod-circulation-storagecourses	36	mod-circulation-storagecourses:171.24.010	2	28801024	2592896	15360	1814768	38488	512128
mod-circulation-calendaritem	6	mod-calendarcirculation-item:31.10.0	2	1024	896	0	768	88	128
mod-source-record-storage18inventory	3	mod-source-record-storage:5.8.7inventory:20.2.0	2	56002880	50002592	20481024	35001814	384	512
mod-event-configemail	6	mod-event-configemail:21.717.0	2	10242800	8962550	0	7681800	88384	128512
mod-coursespubsub	6	mod-coursespubsub:12.413.100	2	10241536	8961440	0	768922	88384	128512
mod-circulation-item	63	mod-circulation-item:124.02.0	2	10242880	8962592	01536	7681814	88384	128512
mod-di-inventoryconverter-storage	36	mod-di-converter-inventorystorage:202.2.0	2	28801024	2592896	10240	1814768	38488	512128
modedge-emailrtac	63	modedge-emailrtac:12.177.01	2	28001024	2550896	0128	1800768	38488	512128
modedge-pubsuborders	6	modedge-pubsuborders:23.130.0	2	15361024	1440896	0	922768	38488	512128
mod-circulationusers	37	mod-circulationusers:2419.23.01	2	28801024	2592896	15360	1814768	38488	512128
mod-ditemplate-converter-storageengine	6	mod-ditemplate-converter-storageengine:21.220.0	2	1024	896	0	768	88	128
edgemod-patron-rtacblocks	36	edgemod-patron-rtacblocks:21.710.10	2	1024	896	1280	768	88	128
edgemod-ordersaudit	6	edgemod-ordersaudit:32.09.0	2	1024	896	0	768	88	128
modedge-usersfqm	76	modedge-usersfqm:192.30.10	2	1024	896	0	768	88	128
mod-source-templaterecord-enginemanager	67	mod-source-templaterecord-enginemanager:13.209.0-SNAPSHOT.330	2	10245600	8965000	0	7683500	88384	128512
modnginx-patron-blocksedge	64	modnginx-patron-blocksedge:12023.1006.014	2	1024	896	0	7680	880	1280
mod-auditquick-marc	68	mod-quick-auditmarc:25.91.02	1	10242288	8962176	0	7681664	88384	128512
edgenginx-fqmokapi	64	edgenginx-fqmokapi:22023.006.014	2	1024	896	0	7680	880	1280
mod-source-record-manager	7	mod-source-record-manager:3.9.0-SNAPSHOT.330	2	5600	5000	0	3500okapi-b	6	okapi:5.3.0	3	1684	1440	0	922	384	512
nginxmod-edgefeesfines	46	nginxmod-edgefeesfines:202319.061.140	2	1024	896	00	768	0880	128
mod-quickinvoice-marcstorage	86	mod-quickinvoice-marcstorage:5.18.01	2	22881872	21761536	0	16641024	384	512
nginxmod-okapidcb	47	nginxmod-okapidcb:20231.061.140	2	1024	896	0	768	88	128
mod-service-interaction	6	mod-service-interaction:4.0.1	2	2048	1844	0	0	okapi-b	6	okapi1290	384	512
mod-data-export	19	mod-data-export:5.30.043	1	16842048	14401844	0	9220	3840	5120
mod-feesfinespatron	63	mod-feesfinespatron:196.1.0	2	1024	896	0128	768	88	128
mod-invoiceoai-storagepmh	6	mod-invoiceoai-storagepmh:53.813.0	2	18724096	15363690	0	10243076	384	512
modedge-dcbconnexion	76	modedge-dcbconnexion:1.12.0	2	1024	896	0	768	88	128
mod-servicekb-ebsco-interactionjava	6	mod-kb-serviceebsco-interactionjava:4.0.10	2	20481024	1844896	0	1290768	38488	512128
mod-data-export	19	mod-data-export:5.0.4	1	2048	1844	0	0	0	0	mod-patron	3notes	6	mod-patronnotes:65.12.0	2	1024	896	1280	768952	88384	128512
mod-oaidata-export-pmhspring	6	mod-data-oaiexport-pmhspring:3.132.02	1	40962048	36901844	0	30761536	384	512
edgemod-organizations-connexionstorage	6	edgemod-organizations-connexionstorage:14.27.0	2	1024	896	0	768	88	128
mod-kb-ebsco-javalogin	6	mod-kb-ebsco-java:4.0login:7.11.0	2	10241440	8961298	0	768	88384	128512
modpub-notesokapi	64	modpub-notesokapi:52023.206.014	2	1024	896	0	952768	3840	5120
mod-data-exporteusage-springreports	6	mod-dataeusage-export-springreports:32.21.011	2	20481024	1844896	0	1536	384	512
mod-organizations-storage	6	mod-organizations-storage:4.7.0	2	1024	896	0	768	88	128
mod-login	6	mod-login:7.11.0	2	1440	1298	0	768	384	512
pub-okapi	4	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	6	mod-eusage-reports:2.1.1	2	1024	896	0	768	88	128

Methodology/Approach

Tests scenarios were started by JMeter script from load generator.

Baseline QCP1 Environment configuration: Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage, number of tasks to launch for service mod-marc-migrations-b was set zero. Instance type: m6g.2xlarge. Instances count: 10. Database db.r6g.xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).

...

768

88

128

Methodology/Approach

Tests scenarios were started by JMeter script from load generator.

Baseline QCP1 Environment configuration: Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage, number of tasks to launch for service mod-marc-migrations-b was set zero. Instance type: m6g.2xlarge. Instances count: 10. Database db.r6g.xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).

Test 1: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run.
Test 2: The Baseline QCP1 Environment configuration was applied, qcp1_MiniMaster.jmx script was run (Repeat Test 1).
Test 3: The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run.
Test 4: The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run (Repeat Test 3).
Test 5: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run.
Test 6: The Baseline QCP1 Environment configuration was applied, and CPU=2 : The Baseline QCP1 Environment configuration was appliedwas set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run run (Repeat Test 15).
Test 37: The Baseline QCP1 Environment configuration was applied, used , and CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: c7g.large for others services, qcp1_MiniMaster.jmx script was run.
Test 48: The The Baseline QCP1 Environment configuration was applied, used two autoscaling groups, 1st with 3 , and CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for okapi service and for all services, qcp1_MiniMaster.jmx script was run.
Test 9: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for others all services, qcp1_MiniMaster.jmx script was run run (Repeat Test 38).
Test 510: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and ONE autoscaling group with 5 Instance Type: c7g.large for others all services, qcp1_MiniMaster.jmx script was run.
Test 611: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and ONE autoscaling group with 5 Instance Type: c7g.large for others all services, qcp1_MiniMaster.jmx script was run run (Repeat Test 510).
Test 712: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and ONE autoscaling group with 5 Instance Type: c7g.large for others all services, qcp1_MiniMaster.jmx script was run (Repeat Test 11).

Version	Old Version 8	New Version Current
Changes made by	Stanislav Nehrii	Stanislav Nehrii
Saved on	Sep 16, 2024	Sep 25, 2024

Name	Memory GIB	vCPUs
db.r6g.xlarge	32 GB	4 vCPUs

Page Comparison

Versions Compared

Key

Overview

Summary

Test Runs

Test Results

Comparison

Test №1

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Test №2

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Test №3

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

Test №4

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №5

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №6

Instance CPU Utilization

Overview

Summary

Test Runs

Test Results

Comparison

Test №1

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

OpenSearch metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №7 - 8

Top SQL-queries

Test №2

Instance CPU Utilization

Service CPU Utilization

Service Memory Utilization

Kafka metrics

OpenSearch metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №9 - 10

Instance CPU Utilization

DB load

Top SQL-queries