Table of Contents

Overview

...

• Comparing kafka.m5.2xlarge against kafka.m7g.2xlarge instance type
  • The main KPI for the workflows do not differ significantly (Data Import durations and CI/CO response time). During 2 hour CI/CO with Data Import tests the number of requests were similar for both MSK clusters- 287669 in m5 and 287155 in m7g MSK instance type. Duration of data import update job with 25k records is longer for 2 minutes with m7g instance type. 
  • MSK resources utilization. CPU decreased slowly (1% - 6%) with m7g instance type. Memory usage is on the same level.
• Resource utilization
  • Memory utilization didn't differ a lot between two MSK clusters
  • Average DB CPU utilization is 85% during create jobs and 87% during update jobs for tests with different MSK instance types. DB CPU utilized 15% during Check-In/Check-Out period without DI.
  • Average connection count is about 850 connections for create and update jobs with CI/CO and 730 connections for CI/CO without data import for tests with different MSK instance types.
  • MSK instance CPU and Disk utilization is similar in both kafka.m7g.2xlarge and kafka.m5.2xlarge.
  • Deltas for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The rest modules CPU utilization deltas fluctuate under 10%. 

...

• Data Import durations and CI/CO response time do not differ significantly. The number of requests in 2 hour CI/CO with Data Import tests were similar for both MSK clusters- 287669 in m5 and 287155 in m7g MSK instance type.

...

PTF -environment qcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK ptf-mobius-testing2
  
  • 2 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2
• MSK perf-921-g2
  • 2 m7g.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.2.tiered

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2

...

• Populate ptf-mobius-testing2 cluster with topics from tenant cluster
• Run CICO for 2 hours
• After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
• Run alone Data Imports
• Create new kafka cluster
• Populate NEW cluster with topics from tenant cluster
• Run CICO for 2 hours
• After 10 min delay after start of CICO Run DI Create - Export - Update for 5 and 25k
• Run alone Data Imports
• Compare resource utilization of MSK and main KPI for CICO & DI

Additional/Files