Overview

• The purpose of the document is to to identify how ECS-related changes are affecting reindexing on multi tenant environment. Calculate reindex time and size of index.

Recommendations & Jiras

• Original ticket to test: PERF-635. 
• Additional info ElasticSearch Reindex Performance Recommendations
  
• Consider optimization of contributor and instance_subject reindexing

Test Summary

Reindexing on 3 tenants in parallel takes the same time as reindexing on main tenant only. If to start in sequential order from secondary tenants it may take more time for main tenant afterwards (25 minutes more). The reason may be less resource utilization from CPU side (mod-search-44% against 38% and mod-inventory-storage - 20% against 19%).

Comparing interim results of indexing size it was found that the major time during reindexing was taken by contributor and instance_subject indexing (at least 7 hours out of 9 hours 45 min).

Test Runs /Results

Indexing size

Test #1 Here I would like to share interim results during reindexing on 3 tenants. In 30 minutes after start we observe that all instances on secondary tenants have been already indexed except of ocp2_contributor_fs09000002 without 1 record. In 2 hours after the start we observe that docs.count column for instances on main tenant has all records but reindexing goes on with previously mentioned contributor_fs09000000 and instance_subject_fs09000000.

So we may conclude that major of time for reindexing goes to contributor indexing (at least 7 hours out of 9 hours 45 min).

Test #2

Memory Usage

Test #1

mod-search max. 70% during an hour, 60% - during second hour

mod-inventory-storage avr. 28%

Test #2, 3 secondary tenants

mod-search max. 50%

mod-inventory-storage avr. 26%

Test #4 main tenant

mod-search max. 70% during an hour, 60% - during second hour

mod-inventory-storage avr. 28%

Memory consumption

CPU Utilization 

Test #1

mod-search - 44%

mod-inventory-storage - 20%

Test #2, 3 secondary tenants

mod-search - 10%

mod-inventory-storage - 10%

Test #4 main tenant

mod-search - 38%

mod-inventory-storage - 19%

CPU utilization

RDS CPU Utilization 

Test #1

Utilization max. - 72%

Test #2, 3 secondary tenants

Utilization max. - 42%

Test #4 main tenant

Utilization max. - 65%

Open Search KPIs

CPU utilization

Test #1

Max. 89%

Test #2, 3 secondary tenants

Max. 32%

Test #4 main tenant

Max. 88%

Indexing rate

Test #1

Test #2, 3 secondary tenants

Test #4 main tenant

Indexing latency 

Test #1

Test #2, 3 secondary tenants

Test #4 main tenant

Appendix

Infrastructure

PTF -environment ocp2

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
• 2 instances of db.r6g.xlarge database instances, one reader, and one writer 
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Number of instances

ocp2 -10M 

Kafka partitioning

Modules memory and CPU parameters:

2.1.0-SNAPSHOT.108

Methodology/Approach

• Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
• Configure the environment in accordance with Infrastructure parameters  to the one that FSE commonly uses
• Reindex on the new Poppy environment, get the results for indexing time, index size
• Create a comparison of results between sequential order and parallel start.