mod-search: Test Reindexing of Instances (Poppy)
Overview
- The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases.
Recommendations & Jiras
- Original ticket to test: PERF-637. Compare with test results of PERF-430.
- Additional info ElasticSearch Reindex Performance Recommendations
- To solve the issue with reindexing RANCHER-1032
Test Summary
Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.
Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032
After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.
It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.
Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).
Average CPU utilization decreased significantly on Poppy (for mod-search on Orchid - 75% against 20% on Poppy, for mod-inventory-storage on Orchid - 50% against 15% on Poppy).
Max. Indexing latency changed from 19 ms on Orchid to 60 ms on Poppy.
Test Runs /Results
Test # | Test Conditions | Duration | Notes |
Reindexing on Orchid release | 11 hr 10 min |
| |
2. 2023_10_01 10:18 - 20:05 UTC | Reindexing on Poppy release | 9 hours 47 min |
|
3. 2023-10-03 10:33 - 11:33 UTC | 1 hour Failed | ||
4. 2023-10-03 12:08 - 13:05 UTC | 57 min Failed | ||
5. 2023-10-09 17:41 - 03:27 UTC | 9 hours 46 min | ||
6. 2023-10-10 06:05 - 15:43 UTC | 9 hours 38 min |
Indexing size
Test #2
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
green | open | ocp2_authority_fs09000000 | XE9S-Me_StyzkP3XQqPqBA | 4 | 0 | 556239 | 0 | 233.1mb | 233.1mb |
green | open | ocp2_contributor_fs09000000 | N59vfY3YTsOopikzeqU5NA | 4 | 2 | 4629419 | 1300484 | 14.6gb | 4.8gb |
green | open | ocp2_instance_fs09000000 | vCgkqlfuRYyWExbd8o0BoQ | 4 | 2 | 10733729 | 2712 | 122.9gb | 40.9gb |
green | open | ocp2_instance_subject_fs09000000 | l-07qruwRRKpKR2ec8Gylg | 4 | 2 | 4071905 | 1176299 | 19.7gb | 6.5gb |
Test #3 Indexing first time after a new OS cluster was created
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
---|---|---|---|---|---|---|---|---|---|
green | open | .opensearch-observability | kPAJ8TqaR06AQZFYekHeyA | 1 | 2 | 0 | 0 | 624b | 208b |
green | open | ocp2_instance_fs09000000 | 61c7lHERQGab0nWBS6tU-w | 4 | 2 | 10733729 | 0 | 124.5gb | 41.3gb |
green | open | ocp2_instance_subject_fs09000000 | T9QYzSdMS-GFcO4skPIQiw | 4 | 2 | 0 | 0 | 2.4kb | 832b |
green | open | ocp2_contributor_fs09000000 | qPUffL5HQtWNpyScGOA6cA | 4 | 2 | 0 | 0 | 2.4kb | 832b |
green | open | .kibana_1 | 9Q4bvyKCRpiiwcPNcLFs9g | 1 | 2 | 1 | 0 | 15.5kb | 5.1kb |
Test #4
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
---|---|---|---|---|---|---|---|---|---|
green | open | .opensearch-observability | kPAJ8TqaR06AQZFYekHeyA | 1 | 2 | 0 | 0 | 624b | 208b |
green | open | ocp2_instance_fs09000000 | pj6TE-z6SkeewcFbfr98yg | 4 | 2 | 10733729 | 0 | 124.7gb | 42gb |
green | open | ocp2_contributor_fs09000000 | lVNwo3siRVKvTGg0_YVivQ | 4 | 2 | 0 | 0 | 2.4kb | 832b |
green | open | ocp2_instance_subject_fs09000000 | LwlykCF9SrWiu4UV1Spmtw | 4 | 2 | 0 | 0 | 2.4kb | 832b |
green | open | .kibana_1 | 9Q4bvyKCRpiiwcPNcLFs9g | 1 | 2 | 1 | 0 | 15.5kb | 5.1kb |
Test #5
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
green | open | ocp2_instance_fs09000000 | RJFH1FFRTLC2f8tAZuTulg | 4 | 2 | 10733729 | 0 | 123.5gb | 40.9gb |
green | open | ocp2_authority_fs09000000 | 9QX5G1CkSMWTIaD-U2Re2g | 4 | 0 | 556239 | 0 | 238.6mb | 238.6mb |
green | open | ocp2_instance_subject_fs09000000 | 8VxNRdHjRYOC-fB-G6B4ow | 4 | 2 | 4076687 | 736676 | 18.1gb | 6gb |
green | open | ocp2_contributor_fs09000000 | 6pmK601zR3eFwZ02PR0SCg | 4 | 2 | 4636652 | 708574 | 12.6gb | 3.8gb |
Test #6
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
green | open | ocp2_instance_fs09000000 | l-8gxUPaRHu9Rb77krV2Pw | 4 | 2 | 10733729 | 0 | 124gb | 41.2gb |
green | open | ocp2_authority_fs09000000 | 9QX5G1CkSMWTIaD-U2Re2g | 4 | 0 | 556239 | 0 | 238.6mb | 238.6mb |
green | open | ocp2_contributor_fs09000000 | oxx-7teWRieeiHmj2mZPqw | 4 | 2 | 4634992 | 732218 | 12.8gb | 3.8gb |
green | open | ocp2_instance_subject_fs09000000 | OaxNET2SRO2nG1oPeTy61w | 4 | 2 | 4076647 | 472411 | 18.6gb | 6.5gb |
Memory Usage
Poppy
Test #2
mod-search max. 74% during an hour, 64% - during second hour and the rest 7 hours with average 47%.
mod-inventory-storage avr. 26%
Test #3,4
Test #5
mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 47%.
mod-inventory-storage avr. 26%
Test #6
mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 45%.
mod-inventory-storage avr. 26%
Orchid
mod-search avr. 36%
mod-inventory-storage avr. 50%
Memory consumption
Test # | Module | Initial spike | avg. usage |
---|---|---|---|
2. Poppy | mod-search | 74% | 47% |
mod-inventory-storage | 26% | 26% | |
5. Poppy | mod-search | 68% | 47% |
mod-inventory-storage | 24% | 22% | |
6. Poppy | mod-search | 60% | 45% |
mod-inventory-storage | 24% | 21% | |
Orchid | mod-search | 47% | 36% |
mod-inventory-storage | 50% | 50% |
CPU Utilization
Poppy
Test #2
After 2 hour of reindexing utilization of CPU resources decreased to minimum. 1-2% for mod-inventory-storage and less than 1 % for mod-search
Test #3,4
Test #5
Test #6
CPU utilization
Test # | Poppy release | Initial spike | avg. usage |
---|---|---|---|
2 | mod-search | 32% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
mod-inventory-storage | 49% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
5 | mod-search | 30% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
mod-inventory-storage | 50% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
6 | mod-search | 30% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
mod-inventory-storage | 54% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
Orchid | mod-search | 200% | 75% - first 2 hours. Close to 1% after 2 hours of indexing |
mod-inventory-storage | 100% | 50% - first 2 hours. Close to 1% after 2 hours of indexing |
Orchid
RDS CPU Utilization
Poppy
Test #2
Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%
Test #3,4