Table of Contents |
---|
...
Test Summary
- Reindexing process duration for consortuim environment takes:
- 3 hours for 3 tenants in parallel (1.7M instances);
- 2 hours for central tenant reindexing (1.2M instances);
- 1 hour for secondary tenant reindexing (353K instances);
- 20 minutes for secondary tenant reindexing (202K instances).
- Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure
- High CPU utilization is obsered on nginx-okapi during the tests - up to 413% during 3 tenants test.
- CPU utilization for mod-inventory-storage reached 102% during test on central tenant, mod-search CPU utilization was about 13-27% during the tests.
- No memory leaks suspected.
Test Runs /Results
Test # | Instances number | Test Conditions reindexing on Poppy release, consortium environment | Duration * | Notes |
1. 2023-11-28 09:20-12:20 UTC | 1766108 | In parallel: 3 tenants | 3 hours |
|
2. 2023-11-29 08:50 - 10:50 UTC | 1212927 | Sequential: cs00000int | 2 hours | |
3. 2023-11-29 14:05 - 15:05 UTC | 353179 | Sequential: cs00000int_0001 | 1 hour | |
4. 2023-11-2915:22 - 15:42 UTC | 200002 | Sequential: cs00000int_0002 | 20 min |
*Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure
Indexing size
All the data from the tables below were capruted after each test was finished.
In parallel: 3 tenants | |||||||||
---|---|---|---|---|---|---|---|---|---|
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
green | open | pcon_instance_subject_cs00000int | oo6lG2KjRBm68SlF4cQf-A | 4 | 2 | 851506 | 188613 | 3.4gb | 1.1gb |
green | open | pcon_contributor_cs00000int | D9izLpcOQWmqZFJhl6kyyA | 4 | 2 | 855300 | 231525 | 1.8gb | 685mb |
green | open | pcon_authority_cs00000int | 5CbWdgFrQSSBRlcNHPk7-A | 4 | 2 | 0 | 0 | 2.4kb | 832b |
green | open | pcon_instance_cs00000int | 6i5vkM6kRHOOu4ahRqESkA | 4 | 2 | 1204190 | 1498 | 23.7gb | 7.8gb |
...
Infrastructure
PTF-environment pcon
- 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...