| Table of Contents |
|---|
...
- Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
- Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.
- Resource utilization
- Memory utilization didn't differ a lot between two MSK clusters
- Disk consume less resources in both (idle and CICO+DI) scenarios with m7g instance type. CPU utilization the same in idle state and from 3% to 18% less under load with m7g instance type.
- Delta for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The most part of modules CPU utilization deltas fluctuate under 10%.
- Average DB CPU usage for both MSK clusters during data import is 85% during create jobs and 87& during Update jobs. Check-In/Check-Out period without DI - 15%.
- Average connection count for both MSK clusters during data import is about 850 connections for create and update jobs with CI/CO. And 730 connections for CI/CO without data import
- MSK instance CPU and Disk utilization remain on the same level in kafka.m7g.2xlarge against kafka.m5.2xlarge or even decreased.
...
Data Import durations and CI/CO response time do not differ significantly. The number of requests during 2 hour CI/CO with Data Import test stayed the same 287669 in m5 and 287155 in m7g MSK instance type.
| Job Profile | File size | DELTA, DI | DELTA, DI+CICO | DELTA, CI | DELTA, CO |
|---|---|---|---|---|---|
| PTF - Create 2 | 5k | 00:00:34 | 00:00:15 | 0.192 | 0.305 |
| 25k | 00:00:14 | 00:00:41 | 0.006 | 0.023 | |
| PTF - Updates Success - 6 | 5k | 00:00:31 | 00:00:20 | 0.065 | 0.133 |
| 25k | 00:02:06 | 00:02:03 | 0.028 | 0.127 |
Kafka resource utilization comparison table
Disk consume less The m7g instance type consumes fewer resources in both ( idle and CICO+DI ) scenarios with m7g instance type. CPU utilization is almost the same in the idle state and from but it is 3% to 18% less lower under load during CI/CO + DI with the m7g instance type.
| MSK instance: kafka.m5.2xlarge | MSK instance: kafka.m7g.2xlarge | Delta | % | ||
|---|---|---|---|---|---|
| Disk usage | Idle state | ||||
| 1 | 0.474 | 0.356 | -0.118 | ||
| 2 | 0.476 | 0.357 | -0.119 | ||
| Under load | |||||
| 1 | 4.6120204 | 4.35 | -0.26202 | ||
| 2 | 4.611104 | 4.35 | -0.2611 | ||
| CPU usage | Idle state | ||||
| 1 | 6.431249875 | 6.63958275 | 0.208333 | ||
| 2 | 6.1516666 | 6.227083625 | 0.075417 | ||
| Under load | |||||
| CICO | 1 | 13.7625025 | 10.6770835 | -3.08542 | -22.42% |
| 2 | 11.94791625 | 9.87916575 | -2.06875 | -17.31% | |
| CICO+DI | 1 | 38.09166625 | 31.13749875 | -6.95417 | -18.26% |
| 2 | 33.82291125 | 32.53334625 | -1.28957 | -3.81% |
...
Appendix
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- MSK perf-921-g2
- 2 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
...