| Table of Contents |
|---|
...
- Comparing kafka.m7g.2xlarge zookeeper metadata mode against kafka.m7g.2xlarge KRaft metadata mode
- No difference in MSK resource utilization (CPU and Disk) comparing two MSK clusters
- In KRaft mode services utilized less memory in mod-inventory-b on 11%, mod-source-record-storage-b on 9%, and more memory in mod-data-import-b on 14%. For the rest modules the difference was less than 7%
- No significant difference in service CPU utilization:
- Service CPU utilization in Zookeeper mode: mod-inventory-b - 136%, mod-quick-marc-b - 96%, mod-di-converter-storage-b - 100%, nginx-okapi - 88% the rest of modules utilized less than 50%
- Service CPU utilization in KRaft mode: mod-inventory-b - 133%, mod-quick-marc-b - 75%, mod-di-converter-storage-b - 103%, nginx-okapi - 75% the rest of modules utilized less than 50%
- No difference in Check-in/Check-out response time
- Data import durations fluctuate from test to test but work stable without issues
...
| MSK instance: kafka.m7g.2xlarge, metadata mode - ZooKeeper | |||||
|---|---|---|---|---|---|
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | 00:03:05 | 00:02:39 | 0.707 | 1.104 |
| 25k | 00:12:03 | 00:12:08 | 0.718 | 1.129 | |
| PTF - Updates Success - 6 | 5k | 00:03:36 | 00:03:34 | 0.742 | 1.124 |
| 25k | 00:17:05 | 00:17:33 | 0.756 | 1.148 | |
| MSK instance: kafka.m7g.2xlarge, metadata mode - KRaft | |||||
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | 00:02:49 | 00:02:39 | 0.765 | 1.118 |
| 25k | 00:13:31 | 00:12:04 | 0.777 | 1.186 | |
| PTF - Updates Success - 6 | 5k | 00:04:36 | 00:04:31 | 0.706 | 1.095 |
| 25k | 00:24:07 | 00:21:50 | 0.74 | 1.16 | |
...
Appendix
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK perf-921-g2
- 2 m7g.2xlarge brokers in 2 zones
Apache Kafka version 2.8.2.tiered
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- MSK ptf-KRaft-mode
- 2 m7g.2xlarge brokers in 2 zones (total 2 brokers)
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- revision - 26
- metadata mode - KRaft
- Total topics: 1474
- Total partitions: 11909
...