Versions Compared

Version Old Version 2 New Version 3
Changes made by

Stanislav Nehrii

Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

Service CPU Utilization

Here we can see that okapi modules used 42k% CPU power parameter CPU=2 for module.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics




DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics



DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics




DB CPU Utilization

DB CPU was 99% maximum.

...

Service CPU Utilization

Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics






DB CPU Utilization

DB CPU was 99%.

...

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics




DB CPU Utilization

DB CPU was 99%.

...

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics




DB CPU Utilization

DB CPU was 99%.

...

Test №7 - 8

Introduction:  

  • Test 7: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 6).
  • Test 8: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of these tests was to validate the consistency of performance observed in Test 5 and Test 6. This was achieved by repeating the same configuration.

Results: We see performance improvements for Test 8.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics





DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 6010.


DB load


Top SQL-queries



Test №9 - 10

Introduction: 

  • Test 9: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 7 after Terminate Instances r7g.xlarge).
  • Test 10: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 8 after Terminate Instances r7g.xlarge).

Objective: The objective of these tests was to validate the consistency of performance observed in Test 7 and Test 8. This was achieved by repeating the same configuration and applying new random task locations per instance after terminating the r7g.xlarge instances.

Results: Performance result were worse on 50% for several workflows after terminating the r7g.xlarge instances and applying new random task locations per instance.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics






DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 5150.


DB load


Top SQL-queries


...

Test №11

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The goal of this test was to replace three c7g.large instances with one additional r7g.xlarge instance for all services, while retaining three c7g.large instances specifically for the okapi service.

Results: The performance was worse compared to when we used the additional three c7g.large instances for services with smaller workloads.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics





DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 5900.


DB load


Top SQL-queries



Test №12 - 13

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics







DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 6200.


DB load


Top SQL-queries



...

Test №14

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics





DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 5900.


DB load


Top SQL-queries


...

Test №15

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi and mod-authtoken service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: 

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 38k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics





DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 5150.


DB load


Top SQL-queries


...

Test №16 - 17

Introduction: Test 16: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: r7g.2xlarge for others services. Fixed Load (average case) MOBIUS test was run. Test 17: Repeat Test 16 with the same environment configuration. 

Objective: The objective of this test was to reduce the number of instances in the main group from 10 to 5, while using higher capacity r7g.2xlarge instances instead of r7g.xlarge

Results: We observed performance improvements in Test 16 compared to Test 8, and by repeating the same configuration in Test 17, we confirmed these performance improvements.

Instance CPU Utilization


Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.


Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.



Kafka metrics





DB CPU Utilization

DB CPU was 99%.


DB Connections

Max number of DB connections was 6100.


DB load


Top SQL-queries


...

PTF - Baseline MCPT environment configuration

  • 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.4xlarge

    		128 GiB16 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK ptf-KRaft-mode
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


MaxMetaspaceSize
Expand
titlemcpt qcp1 modules memory and CPU parameters

Cluster Resources - mcpt-pvt qcp1-pvt (Thu Sep 12 09:30:08 UTC 2024)


ModuleTask
Def.
Definition RevisionModule VersionTask CountMem Hard LimitMem Soft
limit
LimitCPU
units
UnitsXmx
MetaspaceSize
Metaspace SizeMax Metaspace Size
mod-remote-storage
18
6mod-remote-storage:3.2.0
.1
249204472
1024
03960512512
mod-ncip
12
6mod-ncip:1.14.421024896
128
076888128
mod-finance-storage
14
6mod-finance-storage:8.
5
6.0
4
21024896
1024
070088128
mod-agreements
17
6mod-agreements:
6
7.0.02
0
15921488
128
0000
mod-ebsconet
12
6mod-ebsconet:2.
1
2.
1
0212481024
128
0700128256
edge
mod-
sip2
organizations
12
6
edge
mod-
sip2
organizations:
3
1.
1
9.
1
021024896
128
076888128
mod-consortia
18
4mod-consortia:1.1.0
.3
251364776
1024
04416384512
mod
edge-
organizations
sip2
12
4
mod
edge-
organizations
sip2:
1
3.
8
2.0-SNAPSHOT.20921024896
128
076888128
mod-serials-
settings
management
17
6mod-
settings
serials-management:1.0.
2
02
1024
2480
896
2312
200
0
768
1792
88
384
128
512
edge
mod-
dematic
settings
15
6
edge
mod-
dematic
settings:
2
1.
1
0.
1
3
1
21024896
128
076888128
mod-data-import
15
9mod-data-import:3.1.0
.7
120481844
256
01292384512
edge-dematic6edge-dematic:2.2.011024896076888128
mod-search
18
6mod-search:3.2.0
.5
4
225922480
2048
014405121024
mod-inn-
tags
reach
12
4mod-inn-
tags
reach:3.2.
1
0-SNAPSHOT.
0
862
1024
3600
896
3240
128
0
768
2880
88
512
128
1024
mod-
authtoken
tags
22
6mod-
authtoken
tags:2.
14
2.
1
0
6
2
1440
1024
1152
896
1024
0
922
76888128
edge-courses
12
6edge-courses:1.
3
4.021024896
128
076888128
edge
mod-
inventory
authtoken
6
10
edge
mod-
inventory
authtoken:
1
2.
4
15.
0
12
1024
1440
896
1152
128
0
768
92288128
mod-inventory-update
12
6mod-inventory-update:3.
2
3.
1
021024896
128
076888128
mod-notify
12
6mod-notify:3.
1
2.021024896
128
076888128
mod-configuration
13
6mod-configuration:5.
9
10.02
4
1024896
128
076888128
mod-orders-storage
13
6mod-orders-storage:13.
6
7.0
4
21024896
512
070088128
edge-caiasoft
12
6edge-caiasoft:2.
1
2.021024896
128
076888128
mod-login-saml
15
6mod-login-saml:2.
7
8.
2
021024896
128
076888128
mod-erm-usage-harvester
7
6mod-erm
-usage-harvester:4.4.1
-usage-harvester:4.5.021024896076888128
mod-licenses6mod-licenses:6.0.022480231201792384512
mod-gobi6mod-gobi:2.8.021024896
128
076888128
mod-password-validator
12
6mod-password-validator:3.
1
2.0214401298
128
0768384512
mod-
gobi13mod-gobi:2.7.12102489612876888128mod-licenses15mod-licenses:5.0.22248023121281792
bulk-operations6mod-bulk-operations:2.0.023072260001536384512
mod-fqm-manager
12
6mod-fqm-manager:
1
2.0.
3
121024896
128
076888128
edge-dcb
9
6edge-dcb:1.1.0
.2512
21024896
12876888128mod-bulk-operations18mod-bulk-operations:1.1.923072260010241536384
076888128
mod-graphql
12
6mod-graphql:1.12.
0
1
0
2102489612876888128
mod-finance
12
6mod-finance:4.
8
9.0
4
21024896
128
076888128
mod-erm-usage
7
6mod-erm-usage:4.
6
7.02
1024
2800
896
2550
128
0
768
1800
88
384
128
512
mod-
lists
batch-print
15
7mod-
lists
batch-print:1.1.0
.5
21024896
128
076888128
mod-copycat
12
6mod-copycat:1.
5
6.021024896
128
076888128
mod-
permissions
lists
26
6mod-
permissions
lists:
6
2.
4
0.0
4
2
1684
1024
1544
896
512
0
1024
768
384
88
512
128
mod-entities-links
17
7mod-entities-links:
2
3.0.
4
0225922480
400
0144001024
mod-permissions11mod-permissions:6.5.021684154401024384512
pub-edge
6
4pub-edge:2023.06.1421024896
128
076800
mod-orders
13
6mod-orders:12.
7
8.
1
0
4
22048
1440
1740
1024
01024384512
edge-patron
14
3edge-patron:5.
0
1.0
.158
2102489625676888128
edge-ncip
12
6edge-ncip:1.9.221024896
128
076888128
mod-
users
marc-
bl
migrations
12
2mod-
users
marc-
bl
migrations:
7
1.
6
0.0-SNAPSHOT.5
2
0
1440
2048
1152
1764
512
1024
922
153688128
edge-
ea
inn-
data-export
reach
7
4edge-
ea
inn-
data-export
reach:
4
3.1.
0
1-SNAPSHOT.4521024896076888128
768
mod-users-bl6mod-users-bl:7.7.0214401152092288128
mod-
invoice
oa
13
5mod-
invoice
oa:
5.7.
2.1.0-SNAPSHOT.622
4
1024
1440
896
1152922
0
512
76888
128
256
mod-inventory-storage
19
3mod-inventory-storage:27.1.0
.4
4
24096369020483076384512
mod-invoice8mod-invoice:5.8.1214401152092288128
mod-user-import
12
6mod-user-import:3.8.021024896
128
076888128
mod-sender
12
7mod-sender:1.
11
12.021024896
128
076888128
edge-oai-pmh
15
6edge-oai-pmh:2.
7
9.
1
0215121360
1024
01440384512
mod-data-export-worker
15
6mod-data-export
-worker:3.1.223072280010242048384512mod-rtac12mod-rtac
-worker:3.
5
2.
0
12
1024
3072
896
2800
128
0
768
2048
88
384
128
512
mod-
task-list
rtac
7
3mod-
task-list
rtac:
1
3.
9
6.
2
02102489612876888128
mod-circulation-storage
20
3mod-circulation-storage:17.
1
2.
7
0
4
22880259215361814384512
mod-calendar
13
6mod-calendar:
2
3.
5
1.021024896
128
076888128
mod-source-record-storage
21
18mod-source-record-storage:5.8.7
.5
25600500020483500384512
mod-event-config
12
6mod-event-config:2.
6
7.021024896
128
076888128
mod-courses
12
6mod-courses:1.4.
8
1021024896
128
076888128
mod-circulation-item
9
6mod-circulation-item:1.0.021024896
128
076888128
mod-inventory
19
3mod-inventory:20.
1
2.
6
0
4
22880259210241814384512
mod-email
12
6mod-email:1.
16
17.02
1024
2800
896
2550
128
0
768
1800
88
384
128
512
mod-
circulation
pubsub
20
6mod-
circulation
pubsub:
24
2.13.0
.11
4
2
2880
1536
2592
1440
1536
0
1814
922384512
mod-
pubsub
circulation
16
3mod-
pubsub
circulation:24.2.
11.3
02
1536
2880
1440
2592
1024
1536
922
1814384512
mod-di-converter-storage
17
6mod-di-converter-storage:2.
1
2.
5
021024896
128
076888128
edge-rtac
15
3edge-rtac:2.
6
7.
2
12102489612876888128
edge-orders
15
6edge-orders:
2
3.
9
0.
1
021024896
128
076888128
mod-
template-engine
users
12
7mod-
template-engine
users:
1
19.
19
3.121024896
128
076888128
mod-template-
users
engine
17
6mod-template-
users
engine:
19
1.
2
20.02
4
1024896
128
076888128
mod-patron-blocks
14
6mod-patron-blocks:1.
9
10.021024896
1024
076888128
mod-audit
13
6mod-audit:2.
8
9.021024896
128
076888128
edge-fqm
13
6edge-fqm:
1
2.0.
1
021024896
128
076888128
mod-source-record-manager
18
7mod-source-record-manager:3.9.
7
0-SNAPSHOT.
7
330256005000
2048
03500384512
nginx-edge
6
4nginx-edge:2023.06.1421024896
128
0000
mod-quick-marc
16
8mod-quick-marc:5.1.0
.1
122882176
128
01664384512
nginx-okapi
9
4nginx-okapi:2023.06.1421024896
1536
0000
okapi-b
8
6okapi:5.
1
3.
2
0316841440
1536
0922384512
mod-feesfines
12
6mod-feesfines:19.
0
1.021024896
128
076888128
mod-invoice-storage
13
6mod-invoice-storage:5.
7
8.0
4
218721536
1024
01024384512
edge-users6edge-users:1.2.02102489612876888128
mod-dcb
9
7mod-dcb:1.
0
1.021024896
128
076888128
mod-service-interaction
15
6mod-service-interaction:
3
4.0.
2
1220481844
256
01290384512
mod-data-export
17
19mod-data-export:
4
5.
8
0.
7
41
1024
2048
896
1844
1024
0
768
0
88
0
128
0
mod-patron
12
3mod-patron:6.
0
1.0
4
2102489612876888128
mod-oai-pmh
16
6mod-oai-pmh:3.
12
13.
8
0240963690
2048
03076384512
edge-connexion
12
6edge-connexion:1.
1
2.
1
021024896
128
076888128
mod-kb-ebsco-java
12
6mod-kb-ebsco-java:4.0.021024896
128
076888128
mod-notes
13
6mod-notes:5.
1
2.021024896
128
0952384512
mod-data-export-spring
15
6mod-data-export-spring:3.2.0
.2
120481844
256
01536
384512mod-login12
mod-login:7.10.12144012981024768
384512
mod-organizations-storage
13
6mod-organizations-storage:4.
6
7.0
4
21024896
128
076888128
pub
mod-
okapi
login6
pub
mod-
okapi
login:
2023
7.
06
11.
14
02
1024
1440
896
1298
128
0768
0
384
0
512
edge
pub-
erm
okapi
7
4
edge
pub-
erm
okapi:
1
2023.
0
06.
0
1421024896
128
0768
88
0
128
0
mod-eusage-reports
7
6mod-eusage-reports:2.
0
1.
0
121024896
128
076888128




Methodology/Approach

Tests scenarios were started by JMeter script from load generator.   

Baseline QCP1 Environment configuration: Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage,  number of tasks to launch for service mod-marc-migrations-b was set zero. Instance type: m6g.2xlarge. Instances count: 10. Database db.r6g.xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).

...