/
mod-search: Test Reindexing of Instances (Poppy)

mod-search: Test Reindexing of Instances (Poppy)

Oct 16, 2023

Overview

  • The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases. 

Recommendations & Jiras

Test Summary

Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.

Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032

After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.

It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.

Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).

Average CPU utilization decreased significantly on Poppy (for mod-search on Orchid - 75% against 20% on Poppy, for mod-inventory-storage on Orchid - 50% against 15% on Poppy). 

Max. Indexing latency changed from 19 ms on Orchid to 60 ms on Poppy.

Test Runs /Results

Test #

Test Conditions

Duration 

Notes

 

 

Reindexing on Orchid release

11 hr 10 min

  • mod-search task count =8

  • open search instance scaled up to r6g.4xlarge.search

2. 2023_10_01 10:18 - 20:05 UTC

Reindexing on Poppy release

 

 

9 hours 47 min

  • mod-search task count =8

  • mod-inventory-storage task count = 2

  • mod-okapi task count = 3

  • open search instance scaled up to r6g.4xlarge.search

  • without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch

3. 2023-10-03 10:33 - 11:33 UTC

1 hour Failed

4. 2023-10-03 12:08 -  13:05 UTC

57 min Failed

5. 2023-10-09 17:41 - 03:27 UTC

9 hours 46 min

6. 2023-10-10 06:05 - 15:43 UTC

9 hours 38 min

Indexing size

Test #2

health

status

index

uuid

pri 

rep

docs.count

docs.deleted

store.size

pri.store.size

green

open

ocp2_authority_fs09000000

XE9S-Me_StyzkP3XQqPqBA

4

0

556239

0

233.1mb

233.1mb

green

open

ocp2_contributor_fs09000000

N59vfY3YTsOopikzeqU5NA

4

2

4629419

1300484

14.6gb

4.8gb

green

open

ocp2_instance_fs09000000

vCgkqlfuRYyWExbd8o0BoQ

4

2

10733729

2712

122.9gb

40.9gb

green

open

ocp2_instance_subject_fs09000000

l-07qruwRRKpKR2ec8Gylg

4

2

4071905

1176299

19.7gb

6.5gb

Test #3 Indexing first time after a new OS cluster was created

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

green

open

.opensearch-observability

kPAJ8TqaR06AQZFYekHeyA

1

2

0

0

624b

208b

green

open

ocp2_instance_fs09000000

61c7lHERQGab0nWBS6tU-w

4

2

10733729

0

124.5gb

41.3gb

green

open

ocp2_instance_subject_fs09000000

T9QYzSdMS-GFcO4skPIQiw

4

2

0

0

2.4kb

832b

green

open

ocp2_contributor_fs09000000

qPUffL5HQtWNpyScGOA6cA

4

2

0

0

2.4kb

832b

green

open

.kibana_1

9Q4bvyKCRpiiwcPNcLFs9g

1

2

1

0

15.5kb

5.1kb

Test #4

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

green

open

.opensearch-observability

kPAJ8TqaR06AQZFYekHeyA

1

2

0

0

624b

208b

green

open

ocp2_instance_fs09000000

pj6TE-z6SkeewcFbfr98yg

4

2

10733729

0

124.7gb

42gb

green

open

ocp2_contributor_fs09000000

lVNwo3siRVKvTGg0_YVivQ

4

2

0

0

2.4kb

832b

green

open

ocp2_instance_subject_fs09000000

LwlykCF9SrWiu4UV1Spmtw

4

2

0

0

2.4kb

832b

green

open

.kibana_1

9Q4bvyKCRpiiwcPNcLFs9g

1

2

1

0

15.5kb

5.1kb

Test #5

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

green

open

ocp2_instance_fs09000000

RJFH1FFRTLC2f8tAZuTulg

4

2

10733729

0

123.5gb

40.9gb

green

open

ocp2_authority_fs09000000

9QX5G1CkSMWTIaD-U2Re2g

4

0

556239

0

238.6mb

238.6mb

green

open

ocp2_instance_subject_fs09000000

8VxNRdHjRYOC-fB-G6B4ow

4

2

4076687

736676

18.1gb

6gb

green

open

ocp2_contributor_fs09000000

6pmK601zR3eFwZ02PR0SCg

4

2

4636652

708574

12.6gb

3.8gb

Test #6

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

green

open

ocp2_instance_fs09000000

l-8gxUPaRHu9Rb77krV2Pw

4

2

10733729

0

124gb

41.2gb

green

open

ocp2_authority_fs09000000

9QX5G1CkSMWTIaD-U2Re2g

4

0

556239

0

238.6mb

238.6mb

green

open

ocp2_contributor_fs09000000

oxx-7teWRieeiHmj2mZPqw

4

2

4634992

732218

12.8gb

3.8gb

green

open

ocp2_instance_subject_fs09000000

OaxNET2SRO2nG1oPeTy61w

4

2

4076647

472411

18.6gb

6.5gb

Memory Usage