Overview

The purpose of the document is to assess reindexing performance on a consortium environment. Calculate reindex time and size of index.

Recommendations & Jiras

• Original ticket to test: PERF-635 - Getting issue details... STATUS
• Additional info ElasticSearch Reindex Performance Recommendations

Test Summary

• Reindexing process for consortuim environment takes:
  • 3 hours for 3 tenants in parallel (1.7M instances);
  • 2 hours for central tenant reindexing (1.2M instances);
  • 1 hour for secondary tenant reindexing  (353K instances);
  • 20 minutes for secondary tenant reindexing (202K instances).
• Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure
• High CPU utilization is obsered on nginx-okapi module - up to 413% during 3 tenants test.
• CPU utilization for mod-inventory-storage reached 102% during test on central tenant. mod-search CPU utilization was about 13-27% during all the tests.
• No memory leaks suspected.

Test Runs /Results

*Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure

Indexing size

All the data from the tables below were capruted after each test was finished.

Results from get-request for reindex monitoring:

Results from get-request for reindex monitoring:

Service CPU Utilization

Test #1 (3 tenants in parallel)

Maximum CPU utilization:

nginx-okapi - 413%

mod-inventory-storage - 95%

okapi - 73%

mod-search - 27%

Test #2 (cs00000int main tenant, sequental)

Maximum CPU utilization:

nginx-okapi - 332%

mod-inventory-storage - 102%

okapi - 63%

mod-search - 27%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum CPU utilization:

nginx-okapi - 285%

mod-inventory-storage - 98%

okapi - 63%

mod-search - 13%

Memory Utilization

Test #1 (3 tenants in parallel)

Memory utilization:

mod-search - 38% → 50%

mod-inventory-storage - 11% → 31%

Test #2 (cs00000int main tenant, sequental)

Memory utilization:

mod-search - 37% → 50%

mod-inventory-storage - 13% → 21%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Memory utilization:

mod-search - 33% → 49%

mod-inventory-storage - 26% → 31%

DB CPU Utilization

Test #1 (3 tenants in parallel)

Maximum DB CPU utilization - 37%

Test #2 (cs00000int main tenant, sequental)

Maximum DB CPU utilization - 56%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum DB CPU utilization - 36%

DB Connections

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Open Search CPU

Test #1 (3 tenants in parallel)

Maximum CPU utilization - 57%

Test #2 (cs00000int main tenant, sequental)

Maximum CPU utilization - 47%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Maximum CPU utilization - 53%

Open Search Ibdexing Data Rate

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Open Search Ibdexing Latency

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

Appendix

Infrastructure

PTF-environment pcon

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
• 2 instances of db.r6g.xlarge database instances, one reader and one writer 
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Data structure

Methodology/Approach

• Use consortium cluster for testing (pcon in our case).
• Configure the environment in accordance with Infrastructure parameters  to the one that FSE commonly uses.
• Run reindex, get the results for indexing time, index size. Use Steps for testing process#Reindex for details.
• Compare results.