Table of Contents

...

Infrastructure

PTF -environment (ncp1)

• 9 m6i.2xlarge EC2 instances located in us-west-2. 
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK (ptf-kafka-1 cluster)
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=480
  • default.replication.factor=3
  • Apache Kafka v2.8.0
  • EBS storage volume per broker = 300GB
  • Kafka topics 
    • .data-export.job.command  - 50 partitions
    • data-export.job.update - 50 partitions

...

As seen, all the jobs were passing successfully. Some got started later than the other.  Most of the time a bunch of jobs got started first at the same time, then after that each remaining job takes turn to get started.  This is evident in tests 3-6.  Another thing to note is the duration of each job is consistent no matter the number of concurrent jobs being thrown at it.  

...