| Table of Contents |
|---|
...
| MSK instance: kafka.m5.2xlarge | |||||
|---|---|---|---|---|---|
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | 00:03:45 | 00:02:44 | ||
| 25k | 00:14:40 | 00:13:36 | |||
| PTF - Updates Success - 6 | 5k | 00:04:43 | 00:04:18 | ||
| 25k | 00:20:21 | 00:21:25 | |||
| MSK instance: kafka.m7g.2xlarge | |||||
| Job profile | File size | DI Duration without CI/CO | DI Duration with CI/CO | CI with DI Average sec | CO with DI Average sec |
| PTF - Create 2 | 5k | ||||
| 25k | |||||
| PTF - Updates Success - 6 | 5k | ||||
| 25k | |||||
...
- Data Import durations and CI/CO response time do not differ significantly. The number of requests in 2 hour CI/CO with Data Import tests were similar for both MSK clusters- 287669 in m5 and 287155 in m7g MSK instance type.
| Job Profile | File size | DELTA, DI without CI/CO | DELTA, DI+CI/CO | DELTA, CI with DI | DELTA, CO with DI |
|---|---|---|---|---|---|
| PTF - Create 2 | 5k | ||||
| 25k | |||||
| PTF - Updates Success - 6 | 5k | ||||
| 25k |
...
Appendix
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- revision - 2
- MSK ptf-KRaft-mode
- 2 m7g.2xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- revision - 26
...