Table of Contents outline true
Overview
- The purpose of the document is to to identify how ECS-related changes are affecting reindexing on multi tenant environment. Calculate reindex time and size of index.
Recommendations & Jiras
- Original ticket to test: PERF-635.
- Additional info ElasticSearch Reindex Performance Recommendations
- Consider optimization of contributor and instance_subject reindexing
Test Summary
Reindexing on 3 tenants in parallel takes the same time as reindexing on main tenant only. If to start in sequential order from secondary tenants it may take more time for main tenant afterwards (25 minutes more). The reason may be less resource utilization from CPU side (mod-search-44% against 38% and mod-inventory-storage - 20% against 19%).
Comparing interim results of indexing size it was found that the major time during reindexing was taken by contributor and instance_subject indexing (at least 7 hours out of 9 hours 45 min).
Test Runs /Results
Test # | Instances number | Test Conditions reindexing on Poppy release | Duration | Notes |
1. 2023_10_12 08:47 - 18:32 UTC | In parallel: 3 tenants | 9 hours 45 min |
| |
2. 2023-10-13 09:04 - 09:07 UTC | 100032 | In sequential: fs09000002 | 3 min | |
3. 2023-10-13 09:15 - 09:18 UTC | 100055 | In sequential: fs09000003 | 3 min | |
| 4. 2023-10-13 09:29 - 19:37 UTC | 10,733,729 | In sequential: fs09000000 | 10 hours 8 min |
Indexing size
Test #1 Here I would like to share interim results during reindexing on 3 tenants. In 30 minutes after start we observe that all instances on secondary tenants have been already indexed except of ocp2_contributor_fs09000002 without 1 record. In 2 hours after the start we observe that docs.count column for instances on main tenant has all records but reindexing goes on with previously mentioned contributor_fs09000000 and instance_subject_fs09000000.
...
| fs09000002 | |||||||||
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_instance_fs09000002 | Pd2I7_Q8Rj2bNgSP4vSzIg | 4 | 2 | 100032 | 0 | 579mb | 190.3mb |
| green | open | ocp2_authority_fs09000002 | vFMI3x8bTuqp5HkAL3jk8A | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | ocp2_instance_subject_fs09000002 | 3vfuyqMBRlGtmblChed_LA | 4 | 2 | 94 | 5 | 413.9kb | 136.5kb |
| green | open | ocp2_contributor_fs09000002 | xgXcKSDWR3W2IYC6JiFqSA | 4 | 2 | 369 | 1238 | 696.8mb | 165.4mb |
| Test #3 | |||||||||
| fs09000003 | |||||||||
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_instance_fs09000003 | YjbGBqXpRG6ap03WlRFokw | 4 | 2 | 100055 | 0 | 579.8mb | 190.8mb |
| green | open | ocp2_authority_fs09000003 | RGz9CTCoT7CoT3H3ge5gIA | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | ocp2_contributor_fs09000003 | O4SdQSXqT_y2UapCuer6GQ | 4 | 2 | 372 | 61 | 853mb | 259.1mb |
| green | open | ocp2_instance_subject_fs09000003 | 2P6hQELJTWKYqvJ5l-5jTw | 4 | 2 | 98 | 6 | 413.6kb | 136.3kb |
| Test #4 | |||||||||
| fs09000000 | |||||||||
| health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
| green | open | ocp2_instance_fs09000000 | VIviDunkSkinkiZJuMjKUQ | 4 | 2 | 10733729 | 0 | 119.6gb | 39.9gb |
| green | open | ocp2_authority_fs09000000 | YCS4y0GFTfy3EpbJm99X0g | 4 | 2 | 0 | 0 | 2.4kb | 832b |
| green | open | ocp2_contributor_fs09000000 | 2kdpF4fZTaGtFmG_ypXq6A | 4 | 2 | 4666659 | 612503 | 12.2gb | 4.1gb |
| green | open | ocp2_instance_subject_fs09000000 | q3wktVUgTsmElTq6Iv-p2A | 4 | 2 | 4097011 | 658839 | 17.8gb | 6.1gb |
Memory Usage
Test #1
mod-search max. 70% during an hour, 60% - during second hour
...
| Test # | Module | Initial spike |
|---|---|---|
1 | mod-search | 70% |
| mod-inventory-storage | 28% | |
| 2 | mod-search | 50% |
| mod-inventory-storage | 26% | |
| 3 | mod-search | 50% |
| mod-inventory-storage | 26% | |
| 4 | mod-search | 70% |
| mod-inventory-storage | 28% |
CPU Utilization
Test #1
mod-search - 44%
mod-inventory-storage - 20%
...
| Test # | Module | Initial spike |
|---|---|---|
1 | mod-search | 44% |
| mod-inventory-storage | 20% | |
| 2 | mod-search | 10% |
| mod-inventory-storage | 10% | |
| 3 | mod-search | 10% |
| mod-inventory-storage | 10% | |
| 4 | mod-search | 38% |
| mod-inventory-storage | 19% |
RDS CPU Utilization
Test #1
Utilization max. - 72%
...
Test #4 main tenant
Utilization max. - 65%
Open Search KPIs
CPU utilization
Test #1
Max. 89%
...
Test #2, 3 secondary tenants
Test #4 main tenant
Appendix
Infrastructure
PTF -environment ocp2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-search Poppy | 3.0.0-SNAPSHOT.156 | 6 | 8 | 2048 | 2592 | 2480 | 512 | 1440 |
| mod-inventory-storage Poppy | 26.1.0-SNAPSHOT.685 | 11 | 2 | 2048 | 4096 | 3690 | 512 | 3076 |
| mod-search Orchid | 2.1.0-SNAPSHOT.108 | 3 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
| mod-inventory-storage Orchid | 26.1.0-SNAPSHOT.644 | 3 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
Methodology/Approach
- Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
- Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses
- Reindex on the new Poppy environment, get the results for indexing time, index size
- Create a comparison of results between sequential order and parallel start.
...