Table of Contents outline true
...
- Through a series of experiments involving different placement strategies, instance types, and total instance counts, we found that the performance remained consistent when using six
c7g.largeinstances (three dedicated to theokapiservice and three allocated tomod-courses,mod-sender,mod-tasks-list,mod-gobi,edge-dematic,mod-erm-usage,mod-eusage-reports,mod-notify, andmod-data-importservices) alongside tenr7g.xlargeinstances for all other services, with the CPU parameter set to 2 for all services. Notably, this environment configuration offers a 40% cost reduction compared to the existing setup, making it a more economical option without compromising on performance. - Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.
Test Runs
| Test # | Description | Status |
|---|---|---|
| Test 1 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: r7g.xlarge for others services. | Completed |
| Test 2 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others modules. | Completed |
| Test 3 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others services (Repeat Test 1). | Completed |
| Test 4 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. | Completed |
| Test 5 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules (Repeat Test 4). | Completed |
| Test 6 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows. | Completed |
| Test 7 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows (Repeat Test 6). | Completed |
| Test 8 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. | Completed |
| Test 9 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows (Repeat Test 7 after Terminate Instances r7g.xlarge). | Completed |
| Test 10 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules (Repeat Test 8 after Terminate Instances r7g.xlarge). | Completed |
| Test 11 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. | Completed |
| Test 12 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11). | Completed |
| Test 13 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11). | Completed |
| Test 14 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11 after terminate 11 instances). | Completed |
| Test 15 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi and mod-authtoken services and 11 Instance Type: r7g.xlarge for others services. | Completed |
Test Results
This table contains durations for all Workflows.
...
Service CPU Utilization
Here we can see that okapi modules used 42k% CPU power parameter CPU=2 for module.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% average with ERW: Exporting Receiving Information.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% average with ERW: Exporting Receiving Information
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% maximum.
...
Service CPU Utilization
Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
...
Service CPU Utilization
Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
...
Service CPU Utilization
Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
...
Test №7 - 8
Introduction:
- Test 7: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 6).
- Test 8: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.
Objective:
Results:
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 6010.
DB load
Top SQL-queries
Test №9 - 10
Introduction:
- Test 9: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 7 after Terminate Instances r7g.xlarge).
- Test 10: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 8 after Terminate Instances r7g.xlarge).
Objective: The objective of these tests was to validate the consistency of performance observed in Test 7 and Test 8. This was achieved by repeating the same configuration and applying new random task locations per instance after terminating the r7g.xlarge instances.
Results: Performance result were worse on 50% for several workflows after terminating the r7g.xlarge instances and applying new random task locations per instance.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5150.
DB load
Top SQL-queries
...
Test №11
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The goal of this test was to replace three c7g.large instances with one additional r7g.xlarge instance for all services, while retaining three c7g.large instances specifically for the okapi service.
Results: The performance was worse compared to when we used the additional three c7g.large instances for services with smaller workloads.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
Test №12
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.
Results: We confirmed performance degradation with this configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
Test №13
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.
Results: We confirmed performance degradation with this configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
Test №14
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.
Results: We confirmed performance degradation with this configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
...
Test №15
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.
Results: We confirmed performance degradation with this configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
...
PTF - Baseline MCPT environment configuration
- 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs db.r6g.4xlarge
128 GiB 16 vCPUs - Open Search ptf-test
- Data nodes
- Instance type - r6g.2xlarge.search
- Number of nodes - 4
- Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
- Instance type - r6g.large.search
- Number of nodes - 3
- Data nodes
- MSK ptf-KRaft-mode
- 2 brokers, kafka.m7g.xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Baseline MCPT Environment configuration according to tunning environment from previous report: task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).
...