| Table of Contents |
|---|
...
Test Summary
- Reindexing process for consortuim consortium environment takes:
- 3 hours for 3 tenants in parallel (1.7M instances);
- 2 hours for central tenant reindexing (1.2M instances);
- 1 hour for secondary tenant reindexing (353K instances);
- 20 minutes for secondary tenant reindexing (202K instances).
- Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure
- High CPU utilization is obsered observed on nginx-okapi module - up to 413% during 3 tenants test.
- CPU utilization for mod-inventory-storage reached 102% during test on central tenant. mod-search CPU utilization was about 13-27% during all the tests.
- No memory leaks suspected.
...
Test # | Instances number | Test Conditions reindexing on Poppy release, consortium environment | Duration * | Notes |
1. 2023-11-28 09:20-12:20 UTC | 1766108 | In parallel: 3 tenants | 3 hours |
|
2. 2023-11-29 08:50 - 10:50 UTC | 1212927 | Sequential: cs00000int | 2 hours | |
3. 2023-11-29 14:05 - 15:05 UTC | 353179 | Sequential: cs00000int_0001 | 1 hour | |
| 4. 2023-11-2915:22 - 15:42 UTC | 200002 | Sequential: cs00000int_0002 | 20 min |
...
Test #2 (cs00000int main tenant, sequentalsequential)
Maximum CPU utilization:
...
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Maximum CPU utilization:
...
Test #2 (cs00000int main tenant, sequentalsequential)
Memory utilization:
mod-search - 37% → 50%
...
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Memory utilization:
mod-search - 33% → 49%
...
Test #2 (cs00000int main tenant, sequentalsequential)
Maximum DB CPU utilization - 56%
...
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Maximum DB CPU utilization - 36%
...
Test #2 (cs00000int main tenant, sequentalsequential)
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Open Search CPU
Test #1 (3 tenants in parallel)
...
Test #2 (cs00000int main tenant, sequentalsequential)
Maximum CPU utilization - 47%
...
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Maximum CPU utilization - 53%
Open Search
...
Indexing Data Rate
Test #1 (3 tenants in parallel)
...
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Open Search
...
Indexing Latency
Test #1 (3 tenants in parallel)
...
Test #2 (cs00000int main tenant, sequentalsequential)
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequentalsequential)
Appendix
Infrastructure
PTF-environment pcon
- 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
- Use consortium cluster for testing (pcon in our case).
- Configure the environment in accordance with Infrastructure parameters parameters to the one that FSE commonly uses.
- Run reindex, get the results for indexing time, index size. Use Steps for testing process#Reindex for details.
- Compare results.