Overview

The purpose of this ticket is to check and compare performance of CheckIn CheckOut workflow with and without shared kafka topic.

• The approach is to enable (or add) parameter on mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage task definitions: {
            "name": "KAFKA_PRODUCER_TENANT_COLLECTION",
            "value": "ALL"
          },
• run a test with 8 users and 20 users. 
• Disable (delete) parameter from task definition.
• Rerun.

Need to compare response time of CI/CO for tests with and without shared kafka topic, compare resource usage on service level, DB resource usage and kafka metrics

Summary

• CheckIn CheckOut  workflow did passed with no unexpected errors, tested with and without shared kafka topics;
• There is slight differences in response times with and without shared Kafka topic. However with 20 concurrent users differences is small (47 ms. longer for CI/CO with using shared topics) Please see Comparisons table below. 
• Resource usage stats the same for identical tests with and without shared topics. (CPU, Memory , RDS CPU ,DB Connections);
• Kafka usage is barely visible. Possibly we'll achieve more kafka usage with multi tenant CICO with and without shared topics. 

Recommendations & Jiras (Optional)

Original story: PERF-717

Results

Response Times (Average of all tests listed above, in seconds)

Comparisons

comparison between tests with and without shared kafka topics;

Memory Utilization

Notes:

• no sign of memory leaks on related modules. 
• Drop of memory usage in the middle of a chart is due to modules restart to disable using shared Kafka topics

CPU Utilization 

Note: CPU utilization is the same for all of a modules with no dependency if CICO is using shared Kafka topics or not.

RDS CPU Utilization 

Note: 

RDS CPU usage and connections rate is the same for all tests (with and without using shared Kafka topics)

Kafka metrics

Note: We've provided here all charts related to MSK (kafka) cluster. tests here is barely visible. Possibly it will be visible when we'll tests CICO with multiple tenants using shared Kafka topics.

Results Mobius like environment 

Resources Utilization/comparison 

Appendix

Infrastructure

PTF -environment pcp1

Release - Poppy.

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer db.r6g.xlarge

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Open Search fse-ptf cluster
  • instance type r6g.large.search
  • Data nodes -4
  • Master nodes -3
  • EBS size 500 GiB

Modules version: 

Methodology/Approach

• with the default set up, run 8, 20 users CICO, and collect results; 
• Change task definitions of related modules (mod-inventory, mod-inventory-storage, mod-circluation, mod-circulation-storage,mod-pubsub) to use shared kafka topics. Simply add to the parameters of a service :  {
            "name": "KAFKA_PRODUCER_TENANT_COLLECTION",
            "value": "ALL"
          },
• rerun all the tests and compare results. (Response times of CICO, CPU,Mem utilization, kafka metrics, DB usage)