| Table of Contents |
|---|
...
| MSK instance: kafka.m5.2xlarge, metadata mode - ZooKeeper | |||||
|---|---|---|---|---|---|
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | 00:03:45 | 00:02:44 | 0.736 | 1.16 |
| 25k | 00:14:40 | 00:13:36 | 0.787 | 1.176 | |
| PTF - Updates Success - 6 | 5k | 00:04:43 | 00:04:18 | 0.764 | 1.153 |
| 25k | 00:20:21 | 00:21:25 | 0.767 | 1.179 | |
| MSK instance: kafka.m7g.2xlarge, metadata mode - KRaft | |||||
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | 00:02:49 | 00:02:39 | 0.765 | 1.118 |
| 25k | 00:13:31 | 00:12:04 | 0.777 | 1.186 | |
| PTF - Updates Success - 6 | 5k | 00:04:36 | 00:04:31 | 0.706 | 1.095 |
| 25k | 00:24:07 | 00:21:50 | 0.74 | 1.16 | |
...
| Load scenario | Brokers | MSK instance: kafka.m5.2xlarge | MSK instance: kafka.m7g.2xlarge | Delta, % |
|---|---|---|---|---|
| CICO | 1 | 13 | 9 | -4 |
| 2 | 13 | 9 | -4 | |
| CICO+DI | 1 | 45 | 32 | -13 |
| 2 | 34 | 30 | -4 |
Response time
MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
Service CPU Utilization
...
Average DB CPU utilization is 85% 90% during both create jobs and 87% during update jobs for tests with different MSK instance types. DB CPU utilized 15% during Check-In/Check-Out period without DI.
MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
DB Connections
Average connection count is about 850 900 connections for create and 860 connections for update jobs with CI/CO and 730 . 770 connections for CI/CO without data import for tests with different MSK instance types.
MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
MSK instance resource utilization
| Expand | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||
MSK resource utilization (CPU)
MSK resource utilization (DIsk) was growing gradually during tests with kafka.m5.2xlarge to 10% |
Disk usage by broker
MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
CPU (User) usage by broker
MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
...
MSK instance: kafka.m5.2xlarge
Top SQL-queries:
MSK instance: kafka.m7g.2xlarge
Top SQL-queries:
Appendix
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones (total 2 brokers)
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- revision - 2
- metadata mode - ZooKeeper
- MSK ptf-KRaft-mode
- 2 m7g.2xlarge brokers in 2 zones (total 2 brokers)
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- revision - 26
- metadata mode - KRaft
...
- Populate ptf-mobius-testing2 cluster with topics from tenant cluster
- Run CICO for 2 hours
- After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
- Run alone Data Imports
- Create new kafka cluster
- Populate NEW cluster with topics from tenant cluster
- Run CICO for 2 hours
- After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
- Run alone Data Imports
- Compare resource utilization of MSK and main KPI for CICO & DI
Additional/Files
Topics:
| View file | ||||
|---|---|---|---|---|
|
Excel raw data: