Overview

• The purpose of the document is to to identify how ECS-related changes are affecting reindexing on multi tenant environment. Calculate reindex time and size of index.

Recommendations & Jiras

• Original ticket to test: PERF-635. 
• Additional info ElasticSearch Reindex Performance Recommendations
  

Test Summary

Test Runs /Results

Indexing size

Test #1 Here I would like to share interim results during reindexing on 3 tenants. In 30 minutes after start we observe that all instances on secondary tenants have been already indexed except of ocp2_contributor_fs09000002 without 1 record. In 2 hours after the start we observe that docs.count column for instances on main tenant has all records but reindexing goes on with previously mentioned contributor_fs09000000 and instance_subject_fs09000000.

So we may conclude that major of time for reindexing goes to contributor indexing (at least 7 hours out of 9 hours 45 min).

Test #2, 3, 4

Memory Usage

Test #1

mod-search max. 70% during an hour, 60% - during second hour

mod-inventory-storage avr. 28%

Test #2, 3 secondary tenants

mod-search max. 50%

mod-inventory-storage avr. 26%

Test #4 main tenant

mod-search max. 70% during an hour, 60% - during second hour

mod-inventory-storage avr. 28%

Memory consumption

CPU Utilization 

Test #1

CPU utilization

Orchid

RDS CPU Utilization 

Poppy

Test #2

Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%

Test #3,4

Test #5

Test #6

Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.

Orchid

Open Search KPIs

Indexing rate Poppy

Test #2

Duration: 9 hours 47 min

Test #3,4

Test #5

Test #6

Indexing rate Orchid

Duration: 11 hours 10 min

Indexing latency Poppy

Test #2

After 7 hours of indexing latency grew from 35 ms to 55 ms during 1 minute

Max: 79.2 ms

Avr.: 39.6 ms

Test #3,4

Test #5

Test #6

Indexing latency Orchid

Appendix

Infrastructure

Tests #2,3,4

PTF -environment ocp2

• 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
• 2 instances of db.r6.xlarge database instances, one reader, and one writer 
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 310 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Tests #5,6

PTF -environment ocp2

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
• 2 instances of db.r6g.xlarge database instances, one reader, and one writer 
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Number of instances

ocp2 -10M 

Kafka partitioning

Modules memory and CPU parameters: tests #2,3,4

2.1.0-SNAPSHOT.108

Modules memory and CPU parameters:  tests #5,6

Methodology/Approach

• Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
• Configure the environment in accordance with Infrastructure parameters  to the one that FSE commonly uses
• Reindex on the new Poppy environment, get the results for indexing time, index size
• Compare with results of https://issues.folio.org/browse/PERF-430