CICO with/without shared kafka topics
- Roman_Fedynyshyn
Overview
The purpose of this ticket is to check and compare performance of CheckIn CheckOut workflow with and without shared kafka topic.
- The approach is to enable (or add) parameter on mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage task definitions: {
"name": "KAFKA_PRODUCER_TENANT_COLLECTION",
"value": "ALL"
}, - run a test with 8 users and 20 users.
- Disable (delete) parameter from task definition.
- Rerun.
Need to compare response time of CI/CO for tests with and without shared kafka topic, compare resource usage on service level, DB resource usage and kafka metrics
Summary
- CheckIn CheckOut workflow did passed with no unexpected errors, tested with and without shared kafka topics;
- There is slight differences in response times with and without shared Kafka topic. However with 20 concurrent users differences is small (47 ms. longer for CI/CO with using shared topics) Please see Comparisons table below.
- Resource usage stats the same for identical tests with and without shared topics. (CPU, Memory , RDS CPU ,DB Connections);
- Kafka usage is barely visible. Possibly we'll achieve more kafka usage with multi tenant CICO with and without shared topics.
Recommendations & Jiras (Optional)
Original story: PERF-717
Results
Response Times (Average of all tests listed above, in seconds)
| Check-in-check-out | |||
|---|---|---|---|
| Average (seconds) | |||
| Check-in | Check-out | notes | |
| Test 1 8 users | 0.611s | 1,138s | with shared kafka topic |
| Test 2 20 users | 0.537s | 0.935s | with shared kafka topic |
| Test 3 8 users | 0.528s | 0.959s | without shared kafka topic |
| Test 4 20 users | 0.49s | 0.888s | without shared kafka topic |
Comparisons
comparison between tests with and without shared kafka topics;
| Check-in-check-out | |||
|---|---|---|---|
| Average (seconds) | |||
| Check-in | Check-out | notes | |
| Test 1vs3 (8 users) | +0.083s | +0.179s | with/without shared kafka topics |
| Test 2vs4 (20 users) | +0.047s | +0.047s | with/without shared kafka topics |
Memory Utilization
Notes:
- no sign of memory leaks on related modules.
- Drop of memory usage in the middle of a chart is due to modules restart to disable using shared Kafka topics
CPU Utilization
Note: CPU utilization is the same for all of a modules with no dependency if CICO is using shared Kafka topics or not.
RDS CPU Utilization
Note:
RDS CPU usage and connections rate is the same for all tests (with and without using shared Kafka topics)
Kafka metrics
Note: We've provided here all charts related to MSK (kafka) cluster. tests here is barely visible. Possibly it will be visible when we'll tests CICO with multiple tenants using shared Kafka topics.
Results Mobius like environment
| test | Vusers | CI(avg, s) | CO (avg, s) | |
|---|---|---|---|---|
| 3 tenants no shared topic | 8 | 0,771 | 1,359 | |
| 3 tenants with shared topic | 8 | 0,686 | 1,22 | |
| diff | -0,085 | -0,139 | ||
| 3 tenants no shared topic | 20 | 0,661 | 1,187 | |
| 3 tenants with shared topic | 20 | 0,657 | 1,157 | |
| diff | -0,004 | -0,03 | ||
| 10 tenants no shared topic | 8 | 0,7185 | 1,272 | |
| 10 tenants with shared topic | 8 | 0,69 | 1,214 | |
| diff | -0,0285 | -0,058 | ||
| 10 tenants no shared topic | 20 | 0,694 | 1,208 | |
| 10 tenants with shared topic | 20 | 0,703 | 1,25 | |
| diff | 0,009 | 0,042 | ||
Resources Utilization/comparison
| Without shared Kafka topics | With shared Kafka topics | |
|---|---|---|
| CPU usage | CPU usage | |
|
| |
Memory usage | Memory usage | |
|
| |
MSK CPU usage | MSK CPU usage | |
|
|
Appendix
Infrastructure
PTF -environment pcp1
Release - Poppy.
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer db.r6g.xlarge
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Open Search fse-ptf cluster
- instance type r6g.large.search
- Data nodes -4
- Master nodes -3
- EBS size 500 GiB
Modules version:
| Module pcp1-pvt Thu Oct 26 10:24:30 UTC 2023 | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage-b | 11 | mod-inventory-storage:27.0.0 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | false |
| mod-inventory-b | 10 | mod-inventory:20.1.0 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | false |
| mod-circulation-storage-b | 10 | mod-circulation-storage:17.1.0 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | false |
| mod-pubsub-b | 9 | mod-pubsub:2.11.0 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 | false |
| mod-circulation-b | 10 | mod-circulation:24.0.0 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | false |
Methodology/Approach
- with the default set up, run 8, 20 users CICO, and collect results;
- Change task definitions of related modules (mod-inventory, mod-inventory-storage, mod-circluation, mod-circulation-storage,mod-pubsub) to use shared kafka topics. Simply add to the parameters of a service : {
"name": "KAFKA_PRODUCER_TENANT_COLLECTION",
"value": "ALL"
}, - rerun all the tests and compare results. (Response times of CICO, CPU,Mem utilization, kafka metrics, DB usage)