Table of Contents outline true
...
- Original ticket to test: PERF-637. Compare with test results of PERF-430.
- Additional info ElasticSearch Reindex Performance Recommendations
- To solve the issue with reindexing RANCHER-1032
Test Summary
Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.
Incorrect behaviour for reindexing was observed during 3, 4 tests. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032
After problem was fixed additional 2 tests were carried out. So reindexing results during 5, 6 tests are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.
It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with contributor.
Test Runs /Results
Test # | Test Conditions | Duration | Notes |
| Reindexing on Orchid release | 11 hr 10 min |
| |
2. 2023_10_01 10:18 - 20:05 UTC | Reindexing on Poppy release | 9 hours 47 min |
|
3. 2023-10-03 10:33 - 11:33 UTC | 1 hour Failed | ||
4. 2023-10-03 12:08 - 13:05 UTC | 57 min Failed | ||
| 5. 2023-10-09 17:41 - 03:27 UTC | 9 hours 46 min | ||
| 6. 2023-10-10 06:05 - 15:43 UTC | 9 hours 38 min |
...
RDS CPU Utilization
Poppy
2 Test
Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%
3, 4 Tests
5 Test
6 Test
Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.
...
Duration: 9 hours 47 min
3, 4 Tests
5 Test
6 Test
Indexing rate Orchid
...
Max: 79.2 ms
Avr.: 39.6 ms
3, 4 Tests
5 Test
6 Test
Indexing latency Orchid
...
2-4 tests
PTF -environment ocp2
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 310 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
5-6 tests
PTF -environment ocp2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...