Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Overview

...

...

  • Reindexing process for consortium environment takes:
    • 3 hours for 3 tenants in parallel (1.7M instances);
    • 2 hours for central tenant reindexing (1.2M instances);
    • 1 hour for secondary tenant reindexing  (353K instances);
    • 20 minutes for secondary tenant reindexing (202K instances).
  • Duration depends not only on instance number, but also it's its type (source). Data details can be found here: Datastructure
  • High CPU utilization is observed on nginx-okapi module - up to 413% during 3 tenants test.
  • CPU utilization for mod-inventory-storage reached 102% during test on central tenant. mod-search CPU utilization was about 13-27% during all the tests.
  • No memory leaks suspected.

...

Test #

Instances number

Test Conditions

reindexing on Poppy release, consortium environment

Duration *

Notes


1. 2023-11-28 09:20-12:20 UTC

1766108

In parallel: 3 tenants

3 hours
  • mod-search task count =8
  • mod-inventory-storage task count = 2
  • mod-okapi task count = 3
  • open search instance scaled up to r6g.4xlarge.search
  • without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch

2. 2023-11-29 08:50 - 10:50 UTC

1212927

Sequential: cs00000int

2 hours

3. 2023-11-29 14:05 - 15:05 UTC

353179

Sequential: cs00000int_0001

1 hour
4. 2023-11-2915:22 - 15:42 UTC200002Sequential: cs00000int_000220 min

...

Infrastructure

PTF-environment pcon

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 2 instances of db.r6g.xlarge database instances, one reader and one writer 
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...