Table of Contents outline true
...
Recommendations & Jiras
- Original ticket to test: PERF-637. Compare with test results of PERF-430.
- Additional info ElasticSearch Reindex Performance Recommendations
- To solve the issue with reindexing RANCHER-1032
Test Summary
Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.
Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032
After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.
It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.
Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).
...
Test # | Test Conditions | Duration | Notes |
| Reindexing on Orchid release | 11 hr 10 min |
| |
2. 2023_10_01 10:18 - 20:05 UTC | Reindexing on Poppy release | 9 hours 47 min |
|
3. 2023-10-03 10:33 - 11:33 UTC | 1 hour Failed | ||
4. 2023-10-03 12:08 - 13:05 UTC | 57 min Failed | ||
| 5. 2023-10-09 17:41 - 03:27 UTC | 9 hours 46 min | ||
| 6. 2023-10-10 06:05 - 15:43 UTC | 9 hours 38 min |
...
RDS CPU Utilization
Poppy
Test #2
Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%
Test #3,4
Test #5
Test #6
Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.
...
Duration: 9 hours 47 min
Test #3,4
Test #5
Test #6
Indexing rate Orchid
...
Avr.: 39.6 ms
Test #3,4
Test #5
Test #6
Indexing latency Orchid
...
Tests #2,3,4
PTF -environment ocp2
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 310 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Tests #5,6
PTF -environment ocp2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
- Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
- Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses
- Reindex on the new Poppy environment, get the results for indexing time, index size
- Compare with results of https://issuesfolio-org.folioatlassian.orgnet/browse/PERF-430