Table of Contents |
---|
...
- Original ticket to test:
Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-635 - Additional info ElasticSearch Reindex Performance Recommendations
Test Summary
- Reindexing process takes
- 3 hours for 3 tenants in parallel
- 2 hours for central tenant reindexing (1.2M instances)
- 1 hour for secondary tenant (353K instances)
- 20 minutes for secondary tenant (202K instances)
Reindexing on 3 tenants in parallel takes the same time as reindexing on main tenant only. If to start in sequential order from secondary tenants it may take more time for main tenant afterwards (25 minutes more). The reason may be less resource utilization from CPU side (mod-search-44% against 38% and mod-inventory-storage - 20% against 19%).
Comparing interim results of indexing size it was found that the major time during reindexing was taken by contributor and instance_subject indexing (at least 7 hours out of 9 hours 45 min).
Test Runs /Results
Test # | Instances number | Test Conditions reindexing on Poppy release, consortium environment | Duration * | Notes |
1. 2023-11-28 09:20-12:20 UTC | 1766108 | In parallel: 3 tenants | 3 hours |
|
2. 2023-11-29 08:50 - 10:50 UTC | 1212927 | Sequential: cs00000int | 2 hours | |
3. 2023-11-29 14:05 - 15:05 UTC | 353179 | Sequential: cs00000int_0001 | 1 hour | |
4. 2023-11-2915:22 - 15:42 UTC | 200002 | Sequential: cs00000int_0002 | 20 min |
...
mod-search - 27%
Test #2 (cs00000int maint main tenant, sequental)
Maximum CPU utilization:
...
mod-inventory-storage - 11% → 31%
Test #2 (cs00000int maint main tenant, sequental)
Memory utilization:
...
Maximum DB CPU utilization - 37%
Test #2 (cs00000int maint main tenant, sequental)
Maximum DB CPU utilization - 56%
...
Test #1 (3 tenants in parallel)
Test #2 (cs00000int maint main tenant, sequental)
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)
...
Maximum CPU utilization - 57%
Test #2 (cs00000int maint main tenant, sequental)
Maximum CPU utilization - 47%
...
Test #1 (3 tenants in parallel)
Test #2 (cs00000int maint main tenant, sequental)
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)
...
Test #1 (3 tenants in parallel)
Test #2 (cs00000int maint main tenant, sequental)
Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)
...
Infrastructure
PTF-environment pcon
- 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6g.xlarge database instances, one reader and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...