Table of Contents outline true
...
| Cluster | Instance Type | Cost per Month (USD) | Number of Instances | Total Cost per Cluster (USD) |
|---|---|---|---|---|
| QCP1 | m6g.2xlarge | $221.76 | 10 | $2,217.60 |
| MCPT | m6g.2xlarge | $221.76 | 14 | $3,104.64 |
| Optimized Infrastructure Two Auto Scaling Groups | c7g.large | $52.20 | 3 | $1,698.84 |
| r7g.2xlarge | $308.45 | 5 | ||
| Optimized Infrastructure One Auto Scaling Groups | r7g.2xlarge | $308.45 | 5 | $1,542.25 |
...
| Test # | Description | Status |
|---|---|---|
| Test 1 | Instance type: m6g.2xlarge. Instances count: 10. | Completed |
| Test 2 | Instance type: m6g.2xlarge. Instances count: 10 (Repeat Test 1). | Completed |
| Test 3 | Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services. | Completed |
| Test 4 | Used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: m6g.2xlarge for others services (Repeat Test 3). | Completed |
| Test 5 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules. | Completed |
| Test 6 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules (Repeat Test 5). | Completed |
| Test 7 | CPU=2 was set for all modules except CPU=2048 for mod-search, used two autoscaling groups, 1st with 3 Instance Type: c7g.large, 3 of them for okapi service and 5 Instance Type: r7g.xlarge for others modules. | Completed |
| Test 8 | CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for all services. | |
| Test 9 | CPU=2 was set for all modules, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 8). | |
| Test 10 | CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services. | |
| Test 11 | CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 10). | |
| Test 12 | CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services (Repeat Test 11). |
...
Service CPU Utilization
Here we can see that mod-inventory-b modules used 73% CPU.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 97% average.
...
Service CPU Utilization
Here we can see that mod-inventory-b modules used 102% CPU in maximum.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 95% average
...
Service CPU Utilization
Here we can see that mod-inventory-b used 78k% of the CPU.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 97% maximum.
...
Service CPU Utilization
Here we can see that mod-data-export-b used 82k% of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 97%.
...
Service CPU Utilization
Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 94%.
...
Service CPU Utilization
Here we can see that mod-data-export-b used 116k% MAX of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 94%.
...
Service CPU Utilization
Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.
...
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 94%.
...
Test №12
Introduction: The Baseline QCP1 Environment configuration was applied, and CPU=2 was set for all modules except CPU=2048 for mod-search, used ONE autoscaling group with 5 Instance Type: c7g.large for all services, qcp1_MiniMaster.jmx script was run
Objective:
Results:
Instance CPU Utilization
Service CPU Utilization
Here we can see that mod-data-export-b used 36k% AVG of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
OpenSearch metrics
DB CPU Utilization
DB CPU was 94%.
DB Connections
Max number of DB connections was 1265.
DB load
Top SQL-queries
Appendix
Infrastructure
PTF - Baseline QCP1 environment configuration (was changed during testing)
- 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs db.r6g.xlarge
32 GB 4 vCPUs - Open Search ptf-test
- Data nodes
- Instance type - r6g.2xlarge.search
- Number of nodes - 4
- Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
- Instance type - r6g.large.search
- Number of nodes - 3
- Data nodes
- MSK fse-tenant
- 2 brokers, kafka.m7g.xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Baseline QCP1 Environment configuration: Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage, number of tasks to launch for service mod-marc-migrations-b was set zero. Instance type: m6g.2xlarge. Instances count: 10. Database db.r6g.xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).
...