| Table of Contents |
|---|
...
- Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
- The main KPI for the workflows do not differ significantly (Data Import durations and CI/CO response time). During 2 hour CI/CO with Data Import tests the number of requests were similar for both MSK clusters- 287669 in m5 and 287155 in m7g MSK instance type. Duration of data import update job with 25k records is longer for 2 minutes with m7g instance type.
- MSK resources utilization. CPU decreased slowly (1% - 6%) with m7g instance type. Memory usage is on the same level.
- Resource utilization
- Memory utilization didn't differ a lot between two MSK clusters
- Average DB CPU utilization is 85% during create jobs and 87% during update jobs for tests with different MSK instance types. DB CPU utilized 15% during Check-In/Check-Out period without DI.
- Average connection count is about 850 connections for create and update jobs with CI/CO and 730 connections for CI/CO without data import for tests with different MSK instance types.
- MSK instance CPU and Disk utilization is similar in both kafka.m7g.2xlarge and kafka.m5.2xlarge.
- Deltas for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The rest modules CPU utilization deltas fluctuate under 10%.
...
- Data Import durations and CI/CO response time do not differ significantly. The number of requests in 2 hour CI/CO with Data Import tests were similar for both MSK clusters- 287669 in m5 and 287155 in m7g MSK instance type.
| Job Profile | File size | DELTA, DI without CI/CO | DELTA, DI+CI/CO | DELTA, CI with DI | DELTA, CO with DI |
|---|---|---|---|---|---|
| PTF - Create 2 | 5k | 00:00:34 | 00:00:15 | 0.192 | 0.305 |
| 25k | 00:00:14 | 00:00:41 | 0.006 | 0.023 | |
| PTF - Updates Success - 6 | 5k | 00:00:31 | 00:00:20 | 0.065 | 0.133 |
| 25k | 00:02:06 | 00:02:03 | 0.028 | 0.127 |
...
Appendix
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- MSK perf-921-g2
- 2 m7g.2xlarge brokers in 2 zones
Apache Kafka version 2.8.2.tiered
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
...