Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Overview

...

Test Summary

  • Reindexing process takes
    • 3 hours for 3 tenants in parallel
    • 2 hours for central tenant reindexing (1.2M instances)
    • 1 hour for secondary tenant (353K instances)
    • 20 minutes for secondary tenant (202K instances)

Reindexing on 3 tenants in parallel takes the same time as reindexing on main tenant only. If to start in sequential order from secondary tenants it may take more time for main tenant afterwards (25 minutes more). The reason may be less resource utilization from CPU side (mod-search-44% against 38% and mod-inventory-storage - 20% against 19%).

Comparing interim results of indexing size it was found that the major time during reindexing was taken by contributor and instance_subject indexing (at least 7 hours out of 9 hours 45 min).

Test Runs /Results

Test #

Instances number

Test Conditions

reindexing on Poppy release, consortium environment

Duration *

Notes


1. 2023-11-28 09:20-12:20 UTC

1766108

In parallel: 3 tenants

3 hours
  • mod-search task count =8
  • mod-inventory-storage task count = 2
  • mod-okapi task count = 3
  • open search instance scaled up to r6g.4xlarge.search
  • without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch

2. 2023-11-29 08:50 - 10:50 UTC

1212927

Sequential: cs00000int

2 hours

3. 2023-11-29 14:05 - 15:05 UTC

353179

Sequential: cs00000int_0001

1 hour
4. 2023-11-2915:22 - 15:42 UTC200002Sequential: cs00000int_000220 min

...

mod-search - 27%


Test #2 (cs00000int maint main tenant, sequental)

Maximum CPU utilization:

...

mod-inventory-storage - 11% → 31%


Test #2 (cs00000int maint main tenant, sequental)

Memory utilization:

...

Maximum DB CPU utilization - 37%


Test #2 (cs00000int maint main tenant, sequental)

Maximum DB CPU utilization - 56%

...

Test #1 (3 tenants in parallel)

Test #2 (cs00000int maint main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

...

Maximum CPU utilization - 57%


Test #2 (cs00000int maint main tenant, sequental)

Maximum CPU utilization - 47%

...

Test #1 (3 tenants in parallel)

Test #2 (cs00000int maint main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

...

Test #1 (3 tenants in parallel)

Test #2 (cs00000int maint main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequental)

...

Infrastructure

PTF-environment pcon

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 2 instances of db.r6g.xlarge database instances, one reader and one writer 
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...