| Table of Contents |
|---|
...
Circulation log is one of the export workflows that are being tested together
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in us-west-2.
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK (ptf-kafka-1 cluster)
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=480
- default.replication.factor=3
- Apache Kafka v2.8.0
- EBS storage volume per broker = 300GB
- Kafka topics
- .data-export.job.command - 50 partitions
- data-export.job.update - 50 partitions
...
- Circulation log workflow is working stable and without any issues until the number of records to be exported reaches 100K records. After this number system becomes unstable and the job becomes unpredictable. The high load causes high DB CPU usage. Moreover the timestamp of when the job starts changes during that behaviour. See more information here
MODAUD-136
- To manipulate with number of records to export we just change dates in from-to section.
- Most loaded part of the system during this workflow is Database. CPU usage on DB reaching 70% during tests;
- Most loaded service is mod-audit and it CPU level reached 10%;
- We can observe growing of memory usage on mod-data-export-worker (from 54% to 62%). It may need additional investigation in future.
...
Note: ticker to investigate slow query PERF-347
Results
| test # | number of records | time |
|---|---|---|
| 1 | 20K | ±5 min |
| 2 | 40K | ±11 min |
| 3 | 60K | ±17 min |
...
ticket created to investigate memory leak PERF-348
DB CPU Utilization
Failed test
...