Overview

The purpose of the document is to assess reindexing performance on a consortium environment. Calculate reindex time and size of index.

Recommendations & Jiras

• Original ticket to test: PERF-635 - Getting issue details... STATUS
• Additional info ElasticSearch Reindex Performance Recommendations

Test Summary

• Reindexing process takes
  • 3 hours for 3 tenants in parallel
  • 2 hours for central tenant reindexing (1.2M instances)
  • 1 hour for secondary tenant (353K instances)
  • 20 minutes for secondary tenant (202K instances)

Reindexing on 3 tenants in parallel takes the same time as reindexing on main tenant only. If to start in sequential order from secondary tenants it may take more time for main tenant afterwards (25 minutes more). The reason may be less resource utilization from CPU side (mod-search-44% against 38% and mod-inventory-storage - 20% against 19%).

Comparing interim results of indexing size it was found that the major time during reindexing was taken by contributor and instance_subject indexing (at least 7 hours out of 9 hours 45 min).

Test Runs /Results

*Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure

Indexing size

Results from get-request for reindex monitoring:

Results from get-request for reindex monitoring:

Service CPU Utilization

Test #1 (3 tenants in parallel)

Maximum CPU utilization:

nginx-okapi - 413%

mod-inventory-storage - 95%

okapi - 73%

mod-search - 27%

Test #2 (cs00000int main tenant, sequental)

Maximum CPU utilization:

nginx-okapi - 332%

mod-inventory-storage - 102%

okapi - 63%

mod-search - 27%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum CPU utilization:

nginx-okapi - 285%

mod-inventory-storage - 98%

okapi - 63%

mod-search - 13%

Memory Utilization

Test #1 (3 tenants in parallel)

Memory utilization:

mod-search - 38% → 50%

mod-inventory-storage - 11% → 31%

Test #2 (cs00000int main tenant, sequental)

Memory utilization:

mod-search - 37% → 50%

mod-inventory-storage - 13% → 21%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Memory utilization:

mod-search - 33% → 49%

mod-inventory-storage - 26% → 31%

DB CPU Utilization

Test #1 (3 tenants in parallel)

Maximum DB CPU utilization - 37%

Test #2 (cs00000int main tenant, sequental)

Maximum DB CPU utilization - 56%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum DB CPU utilization - 36%

DB Connections

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Open Search CPU

Test #1 (3 tenants in parallel)

Maximum CPU utilization - 57%

Test #2 (cs00000int main tenant, sequental)

Maximum CPU utilization - 47%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum CPU utilization - 53%

Open Search Ibdexing Data Rate

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Open Search Ibdexing Latency

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Appendix

Infrastructure

PTF-environment pcon

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
• 2 instances of db.r6g.xlarge database instances, one reader and one writer 
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Data structure

Methodology/Approach

• Use consortium cluster for testing (pcon in our case).
• Configure the environment in accordance with Infrastructure parameters  to the one that FSE commonly uses.
• Run reindex, get the results for indexing time, index size. Use Steps for testing process#Reindex for details.
• Compare results.