...

Approximately DB CPU usage is ± 60%

Kafka metrics

Image Added

Appendix

Infrastructure

PTF -environment ncp3 

• 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 2 instances of db.r6.xlarge database instances, one reader, and one writer
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: 
  • DI_RAW_RECORDS_CHUNK_READ -2 
  • DI_RAW_RECORDS_CHUNK_PARSED -2
  • DI_PARSED_RECORDS_CHUNK_SAVED -2
  • DI_SRS_MARC_AUTHORITY_RECORD_CREATED -1
  • DI_COMPLETED -2

...