Overview

The purpose of the document is to assess reindexing performance on a consortium environment. Calculate reindex time and size of index.

Recommendations & Jiras

• Original ticket to test: PERF-635 - Getting issue details... STATUS
• Additional info ElasticSearch Reindex Performance Recommendations

Test Summary

• Reindexing process for consortium environment takes:
  • 3 hours for 3 tenants in parallel (1.7M instances);
  • 2 hours for central tenant reindexing (1.2M instances);
  • 1 hour for secondary tenant reindexing  (353K instances);
  • 20 minutes for secondary tenant reindexing (202K instances).
• Duration depends not only on instance number, but also its type (source). cs00000int_0001 tenant has a lot of shared instances (343K) but much less unshared ones (10K) compared to cs00000int_0002 (200K unshared). Data details can be found here: Datastructure
• High CPU utilization is observed on nginx-okapi module - up to 413% during 3 tenants test.
• CPU utilization for mod-inventory-storage reached 102% during test on central tenant. mod-search CPU utilization was about 13-27% during all the tests.
• No memory leaks suspected.

Test Runs /Results

*Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure

Indexing size

All the data from the tables below were capruted after each test was finished.

Results from get-request for reindex monitoring:

Results from get-request for reindex monitoring:

Service CPU Utilization

Test #1 (3 tenants in parallel)

Maximum CPU utilization:

nginx-okapi - 413%

mod-inventory-storage - 95%

okapi - 73%

mod-search - 27%

Test #2 (cs00000int main tenant, sequential)

Maximum CPU utilization:

nginx-okapi - 332%

mod-inventory-storage - 102%

okapi - 63%

mod-search - 27%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Maximum CPU utilization:

nginx-okapi - 285%

mod-inventory-storage - 98%

okapi - 63%

mod-search - 13%

Memory Utilization

Test #1 (3 tenants in parallel)

Memory utilization:

mod-search - 38% → 50%

mod-inventory-storage - 11% → 31%

Test #2 (cs00000int main tenant, sequential)

Memory utilization:

mod-search - 37% → 50%

mod-inventory-storage - 13% → 21%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Memory utilization:

mod-search - 33% → 49%

mod-inventory-storage - 26% → 31%

DB CPU Utilization

Test #1 (3 tenants in parallel)

Maximum DB CPU utilization - 37%

Test #2 (cs00000int main tenant, sequential)

Maximum DB CPU utilization - 56%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Maximum DB CPU utilization - 36%

DB Connections

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequential)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Open Search CPU

Test #1 (3 tenants in parallel