mod-search: Test Reindexing of Instances (Poppy)
- Leonid Kolesnykov
Overview
- The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases.
Recommendations & Jiras
- Original ticket to test: PERF-637. Compare with test results of PERF-430.
- Additional info ElasticSearch Reindex Performance Recommendations
- To solve the issue with reindexing RANCHER-1032
Test Summary
Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.
Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032
After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.
It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.
Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).
Average CPU utilization decreased significantly on Poppy (for mod-search on Orchid - 75% against 20% on Poppy, for mod-inventory-storage on Orchid - 50% against 15% on Poppy).
Max. Indexing latency changed from 19 ms on Orchid to 60 ms on Poppy.
Test Runs /Results
Test # | Test Conditions | Duration | Notes |
| Reindexing on Orchid release | 11 hr 10 min |
| |
2. 2023_10_01 10:18 - 20:05 UTC | Reindexing on Poppy release | 9 hours 47 min |
|
3. 2023-10-03 10:33 - 11:33 UTC | 1 hour Failed | ||
4. 2023-10-03 12:08 - 13:05 UTC | 57 min Failed | ||
| 5. 2023-10-09 17:41 - 03:27 UTC | 9 hours 46 min | ||
| 6. 2023-10-10 06:05 - 15:43 UTC | 9 hours 38 min |
Indexing size
Test #2
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_authority_fs09000000 | XE9S-Me_StyzkP3XQqPqBA | 4 | 0 | 556239 | 0 | 233.1mb | 233.1mb |
| green | open | ocp2_contributor_fs09000000 | N59vfY3YTsOopikzeqU5NA | 4 | 2 | 4629419 | 1300484 | 14.6gb | 4.8gb |
| green | open | ocp2_instance_fs09000000 | vCgkqlfuRYyWExbd8o0BoQ | 4 | 2 | 10733729 | 2712 | 122.9gb | 40.9gb |
| green | open | ocp2_instance_subject_fs09000000 | l-07qruwRRKpKR2ec8Gylg | 4 | 2 | 4071905 | 1176299 | 19.7gb | 6.5gb |
Test #3 Indexing first time after a new OS cluster was created
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
|---|---|---|---|---|---|---|---|---|---|
| green | open | .opensearch-observability | kPAJ8TqaR06AQZFYekHeyA | 1 | 2 | 0 | 0 | 624b | 208b |
| green | open | ocp2_instance_fs09000000 | 61c7lHERQGab0nWBS6tU-w | 4 | 2 | 10733729 | 0 | 124.5gb | 41.3gb |
| green | open | ocp2_instance_subject_fs09000000 | T9QYzSdMS-GFcO4skPIQiw | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | ocp2_contributor_fs09000000 | qPUffL5HQtWNpyScGOA6cA | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | .kibana_1 | 9Q4bvyKCRpiiwcPNcLFs9g | 1 | 2 | 1 | 0 | 15.5kb | 5.1kb |
Test #4
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
|---|---|---|---|---|---|---|---|---|---|
| green | open | .opensearch-observability | kPAJ8TqaR06AQZFYekHeyA | 1 | 2 | 0 | 0 | 624b | 208b |
| green | open | ocp2_instance_fs09000000 | pj6TE-z6SkeewcFbfr98yg | 4 | 2 | 10733729 | 0 | 124.7gb | 42gb |
| green | open | ocp2_contributor_fs09000000 | lVNwo3siRVKvTGg0_YVivQ | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | ocp2_instance_subject_fs09000000 | LwlykCF9SrWiu4UV1Spmtw | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | .kibana_1 | 9Q4bvyKCRpiiwcPNcLFs9g | 1 | 2 | 1 | 0 | 15.5kb | 5.1kb |
Test #5
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_instance_fs09000000 | RJFH1FFRTLC2f8tAZuTulg | 4 | 2 | 10733729 | 0 | 123.5gb | 40.9gb |
| green | open | ocp2_authority_fs09000000 | 9QX5G1CkSMWTIaD-U2Re2g | 4 | 0 | 556239 | 0 | 238.6mb | 238.6mb |
| green | open | ocp2_instance_subject_fs09000000 | 8VxNRdHjRYOC-fB-G6B4ow | 4 | 2 | 4076687 | 736676 | 18.1gb | 6gb |
| green | open | ocp2_contributor_fs09000000 | 6pmK601zR3eFwZ02PR0SCg | 4 | 2 | 4636652 | 708574 | 12.6gb | 3.8gb |
Test #6
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_instance_fs09000000 | l-8gxUPaRHu9Rb77krV2Pw | 4 | 2 | 10733729 | 0 | 124gb | 41.2gb |
| green | open | ocp2_authority_fs09000000 | 9QX5G1CkSMWTIaD-U2Re2g | 4 | 0 | 556239 | 0 | 238.6mb | 238.6mb |
| green | open | ocp2_contributor_fs09000000 | oxx-7teWRieeiHmj2mZPqw | 4 | 2 | 4634992 | 732218 | 12.8gb | 3.8gb |
| green | open | ocp2_instance_subject_fs09000000 | OaxNET2SRO2nG1oPeTy61w | 4 | 2 | 4076647 | 472411 | 18.6gb | 6.5gb |
Memory Usage
Poppy
Test #2
mod-search max. 74% during an hour, 64% - during second hour and the rest 7 hours with average 47%.
mod-inventory-storage avr. 26%
Test #3,4
Test #5
mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 47%.
mod-inventory-storage avr. 26%
Test #6
mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 45%.
mod-inventory-storage avr. 26%
Orchid
mod-search avr. 36%
mod-inventory-storage avr. 50%
Memory consumption
| Test # | Module | Initial spike | avg. usage |
|---|---|---|---|
2. Poppy | mod-search | 74% | 47% |
| mod-inventory-storage | 26% | 26% | |
| 5. Poppy | mod-search | 68% | 47% |
| mod-inventory-storage | 24% | 22% | |
| 6. Poppy | mod-search | 60% | 45% |
| mod-inventory-storage | 24% | 21% | |
| Orchid | mod-search | 47% | 36% |
| mod-inventory-storage | 50% | 50% |
CPU Utilization
Poppy
Test #2
After 2 hour of reindexing utilization of CPU resources decreased to minimum. 1-2% for mod-inventory-storage and less than 1 % for mod-search
Test #3,4
Test #5
Test #6
CPU utilization
| Test # | Poppy release | Initial spike | avg. usage |
|---|---|---|---|
2 | mod-search | 32% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
| mod-inventory-storage | 49% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
5 | mod-search | 30% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
| mod-inventory-storage | 50% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
| 6 | mod-search | 30% | 20% - first 2 hours. Close to 1% after 2 hours of indexing |
| mod-inventory-storage | 54% | 15% - first 2 hours. Close to 1% after 2 hours of indexing | |
| Orchid | mod-search | 200% | 75% - first 2 hours. Close to 1% after 2 hours of indexing |
| mod-inventory-storage | 100% | 50% - first 2 hours. Close to 1% after 2 hours of indexing |
Orchid
RDS CPU Utilization
Poppy
Test #2
Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%
Test #3,4
Test #5
Test #6
Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.
Orchid
Open Search KPIs
Indexing rate Poppy
Test #2
Duration: 9 hours 47 min
Test #3,4
Test #5
Test #6
Indexing rate Orchid
Duration: 11 hours 10 min
Indexing latency Poppy
Test #2
After 7 hours of indexing latency grew from 35 ms to 55 ms during 1 minute
Max: 79.2 ms
Avr.: 39.6 ms
Test #3,4
Test #5
Test #6
Indexing latency Orchid
Appendix
Infrastructure
Tests #2,3,4
PTF -environment ocp2
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 310 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Tests #5,6
PTF -environment ocp2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Number of instances
ocp2 -10M
Kafka partitioning
| Orchid | Poppy | |
|---|---|---|
inventory.item | 50 | 50 |
inventory.instance | 50 | 50 |
inventory.holdings-record | 50 | 50 |
inventory.bound-with | 50 | 50 |
inventory.authorit | 50 | 50 |
search.instance-contributor | 50 | 50 |
Modules memory and CPU parameters: tests #2,3,4
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-search Poppy | 3.0.0-SNAPSHOT.151 | 6 | 8 | 2048 | 2592 | 2480 | 512 | 1440 |
| mod-inventory-storage Poppy | 26.1.0-SNAPSHOT.685 | 11 | 2 | 2048 | 4096 | 3690 | 512 | 3076 |
| mod-search Orchid | 2.1.0-SNAPSHOT.108 | 3 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
| mod-inventory-storage Orchid | 26.1.0-SNAPSHOT.644 | 3 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
Modules memory and CPU parameters: tests #5,6
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-search Poppy | 3.0.0-SNAPSHOT.156 | 13 | 8 | 2048 | 2592 | 2480 | 512 | 1440 |
| mod-inventory-storage Poppy | 26.1.0-SNAPSHOT.685 | 11 | 2 | 2048 | 4096 | 3690 | 512 | 3076 |
Methodology/Approach
- Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
- Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses
- Reindex on the new Poppy environment, get the results for indexing time, index size
- Compare with results of https://folio-org.atlassian.net/browse/PERF-430