Table of Contents

outline	true

Overview

The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases.

Recommendations & Jiras

Original ticket to test: PERF-637. Compare with test results of PERF-430.
Additional info ElasticSearch Reindex Performance Recommendations
To solve the issue with reindexing RANCHER-1032

Test Summary

Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.

Incorrect behaviour for reindexing was observed during 3, 4 tests. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032

After problem was fixed additional 2 tests were carried out. So reindexing results during 5, 6 tests are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.

It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with contributor.

Test Runs /Results

...

Test #

...

Test Conditions

...

Duration

Notes

...

mod-search task count =8
open search instance scaled up to r6g.4xlarge.search

...

2. 2023_10_01 10:18 - 20:05 UTC

Reindexing on Poppy release

...

mod-search task count =8
mod-inventory-storage task count = 2
mod-okapi task count = 3
open search instance scaled up to r6g.4xlarge.search
without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch

...

3. 2023-10-03 10:33 - 11:33 UTC

...

4. 2023-10-03 12:08 - 13:05 UTC

...

Indexing size

2 Test

Table of Contents

outline	true

Overview

The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases.

Recommendations & Jiras

Original ticket to test: PERF-637. Compare with test results of PERF-430.
Additional info ElasticSearch Reindex Performance Recommendations
To solve the issue with reindexing RANCHER-1032

Test Summary

Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.

Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032

After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.

It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.

Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).

Average CPU utilization decreased significantly on Poppy (for mod-search on Orchid - 75% against 20% on Poppy, for mod-inventory-storage on Orchid - 50% against 15% on Poppy).

Max. Indexing latency changed from 19 ms on Orchid to 60 ms on Poppy.

Test Runs /Results

Test #	Test Conditions	Duration	Notes
	Reindexing on Orchid release	11 hr 10 min	mod-search task count =8 open search instance scaled up to r6g.4xlarge.search
2. 2023_10_01 10:18 - 20:05 UTC	Reindexing on Poppy release	9 hours 47 min	mod-search task count =8 mod-inventory-storage task count = 2 mod-okapi task count = 3 open search instance scaled up to r6g.4xlarge.search without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch
3. 2023-10-03 10:33 - 11:33 UTC		1 hour Failed
4. 2023-10-03 12:08 - 13:05 UTC		57 min Failed
5. 2023-10-09 17:41 - 03:27 UTC		9 hours 46 min
6. 2023-10-10 06:05 - 15:43 UTC		9 hours 38 min

Indexing size

Test #2

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	ocp2_authority_fs09000000	XE9S-Me_StyzkP3XQqPqBA	4	0	556239	0	233.1mb	233.1mb
green	open	ocp2_contributor_fs09000000	N59vfY3YTsOopikzeqU5NA	4	2	4629419	1300484	14.6gb	4.8gb
green	open	ocp2_instance_fs09000000	vCgkqlfuRYyWExbd8o0BoQ	4	2	10733729	2712	122.9gb	40.9gb
green	open	ocp2_instance_subject_fs09000000	l-07qruwRRKpKR2ec8Gylg	4	2	4071905	1176299	19.7gb	6.5gb

Test #3 Indexing first time after a new OS cluster was created

health	status	index	uuid

pri

pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	.opensearch-observability	kPAJ8TqaR06AQZFYekHeyA	1	2	0	0	624b	208b
green	open	ocp2_

authority

instance_fs09000000

XE9S

61c7lHERQGab0nWBS6tU-

Me_StyzkP3XQqPqBA

w

4

0

2

556239

10733729

0

233

124.

1mb

5gb

233

41.

1mb

3gb
green	open	ocp2_instance_

contributor

subject_fs09000000

N59vfY3YTsOopikzeqU5NA

T9QYzSdMS-GFcO4skPIQiw

4

2

4629419

0

1300484

0

14

2.

6gb

4kb

4.8gb

832b
green	open	ocp2_

instance

contributor_fs09000000

vCgkqlfuRYyWExbd8o0BoQ

qPUffL5HQtWNpyScGOA6cA

4

2

10733729

0

2712

0

122

2.

9gb

4kb

40.9gb4

832b
green	open

ocp2_instance_subject_fs09000000l-07qruwRRKpKR2ec8Gylg

.kibana_1

9Q4bvyKCRpiiwcPNcLFs9g

1

2

4071905

1

1176299

0

19

15.

7gb

5kb

6.5gb

...

5.1kb

Test #4

health	status	index	uuid	pri	rep	docs.count	store.size	pri.store.size
green	open	.opensearch-observability	kPAJ8TqaR06AQZFYekHeyA	1	2	0	624b	208b
green	open	ocp2_instance_fs09000000	61c7lHERQGab0nWBS6tUpj6TE-wz6SkeewcFbfr98yg	4	2	10733729	124.5gb7gb41.3gb	42gb
green	open	ocp2_instance_subjectcontributor_fs09000000	T9QYzSdMS-GFcO4skPIQiwlVNwo3siRVKvTGg0_YVivQ	4	2	0	2.4kb	832b
green	open	ocp2_instance_contributorsubject_fs09000000	qPUffL5HQtWNpyScGOA6cALwlykCF9SrWiu4UV1Spmtw	4	2	0	2.4kb	832b
green	open	.kibana_1	9Q4bvyKCRpiiwcPNcLFs9g	1	2	1	15.5kb	5.1kb

4 Test #5

health

status

index

uuid

pri

rep

docs.count

docs.deleted

store.size

pri.store.size

greenopen.opensearch-observabilitykPAJ8TqaR06AQZFYekHeyA1200624b208b


green	open	ocp2_instance_fs09000000

pj6TE-z6SkeewcFbfr98yg

RJFH1FFRTLC2f8tAZuTulg

4

2

10733729

0

124

123.

7gb

5gb

42gb

40.9gb
green	open	ocp2_

contributor

authority_fs09000000

lVNwo3siRVKvTGg0_YVivQ

9QX5G1CkSMWTIaD-U2Re2g

4

2

0

556239

0

2

238.

4kb

6mb

832b

238.6mb
green	open	ocp2_instance_subject_fs09000000

LwlykCF9SrWiu4UV1Spmtw

8VxNRdHjRYOC-fB-G6B4ow

4

2

0

4076687

0

736676

2

18.

4kb

1gb

832b1

6gb
green	open

.kibana_19Q4bvyKCRpiiwcPNcLFs9g

ocp2_contributor_fs09000000

6pmK601zR3eFwZ02PR0SCg

4

2

1

4636652

0

708574

15

12.

5kb

6gb

5

3.

1kb

8gb

5 Test #6

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	ocp2_instance_fs09000000

RJFH1FFRTLC2f8tAZuTulg

l-8gxUPaRHu9Rb77krV2Pw

4

2

10733729

0

123.5gb

124gb

40

41.

9gb

2gb
green	open	ocp2_authority_fs09000000	9QX5G1CkSMWTIaD-U2Re2g	4	0	556239	0	238.6mb	238.6mb
green	open	ocp2_

instance_subject

contributor_fs09000000

8VxNRdHjRYOC

oxx-

fB-G6B4ow

7teWRieeiHmj2mZPqw

4

2

4076687

4634992

736676

732218

18

12.

1gb

8gb

6gb

3.8gb
green	open	ocp2_instance_

contributor

subject_fs09000000

6pmK601zR3eFwZ02PR0SCg

OaxNET2SRO2nG1oPeTy61w

4

2

4636652

4076647

708574

472411

12

18.6gb

3

6.

8gb

5gb

...

Memory Usage

...

Poppy

...

Test #2

...

Memory Usage

Poppy

2 Test

Image Removed

mod-search max. 74% during an hour, 64% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

3, 4 Tests

Image Removed

5 Test

Image Removed

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

6 Test

Image Removed

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 45%.

mod-inventory-storage avr. 26%

Orchid

Image Removed

mod-search avr. 36%

mod-inventory-storage avr. 50%

...

Initial spike

...

2. Poppy

...

CPU Utilization

Poppy

2 Test

Image Removed

After 2 hour of reindexing utilization of CPU resources decreased to minimum. 1-2% for mod-inventory-storage and less than 1 % for mod-search

3, 4 Tests

Image Removed

5 Test

Image Removed

6 Test

Image Removed

...

Initial spike

...

2

...

Image Added

mod-search max. 74% during an hour, 64% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

Test #3,4

Image Added

Test #5

Image Added

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

Test #6

Image Added

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 45%.

mod-inventory-storage avr. 26%

Orchid

Image Added

mod-search avr. 36%

mod-inventory-storage avr. 50%

Memory consumption

Test #	Module	Initial spike	avg. usage
2. Poppy	mod-search	74%	47%
2. Poppy	mod-inventory-storage	26%	26%
5. Poppy	mod-search	68%	47%
5. Poppy	mod-inventory-storage	24%	22%
6. Poppy	mod-search	60%	45%
6. Poppy	mod-inventory-storage	24%	21%
Orchid	mod-search	47%	36%
Orchid	mod-inventory-storage	50%	50%

CPU Utilization

Poppy

Test #2

Image Added

After 2 hour of reindexing utilization of CPU resources decreased to minimum. 1-2% for mod-inventory-storage and less than 1 % for mod-search

Test #3,4

Image Added

Test #5

Image Added

Test #6

Image Added

CPU utilization

Test #	Poppy release	Initial spike	avg. usage
2	mod-search	32%	20% - first 2 hours. Close to 1% after 2 hours of indexing
2	mod-inventory-storage	49%	15% - first 2 hours. Close to 1% after 2 hours of indexing
5	mod-search	30%	20% - first 2 hours. Close to 1% after 2 hours of indexing
5	mod-inventory-storage	50%	15% - first 2 hours. Close to 1% after 2 hours of indexing
6	mod-search	30%	20% - first 2 hours. Close to 1% after 2 hours of indexing
6	mod-inventory-storage	54%	15% - first 2 hours. Close to 1% after 2 hours of indexing
Orchid	mod-search	200%	75% - first 2 hours. Close to 1% after 2 hours of indexing
Orchid	mod-inventory-storage

49%

100%	50% - first 2 hours. Close to 1% after 2 hours of indexing

5

mod-search30%mod-inventory-storage50%6mod-search30%

mod-inventory-storage54%

Orchid

RDS CPU Utilization

Poppy

2 Test #2

Image Modified

Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%

3Test #3,4 Tests

Image Modified

5 Test #5

Image Modified6 Test

Test #6

Image Modified

Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.

...

Open Search KPIs

Indexing rate Poppy

2 Test #2

Duration: 9 hours 47 min

3Test #3,4 Tests

5 Test #5

Image Modified

6 Test #6

Image Modified

Indexing rate Orchid

...

Duration: 11 hours 10 min

Indexing latency Poppy

2 Test #2

After 7 hours of indexing latency grew from 35 ms to 55 ms during 1 minute

...

Max: 79.2 ms

Avr.: 39.6 ms

3Test #3,4 Tests

5 Test #5

Image Modified

6 Test #6

Image Modified

Indexing latency Orchid

Appendix

Infrastructure

2-4 testsTests #2,3,4

PTF -environment ocp2

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 310 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

5-Tests #5,6 tests

PTF -environment ocp2

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6g.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Modules memory and CPU parameters: 1-4 tests tests #2,3,4

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-search Poppy	3.0.0-SNAPSHOT.151	6	8	2048	2592	2480	512	1440
mod-inventory-storage Poppy	26.1.0-SNAPSHOT.685	11	2	2048	4096	3690	512	3076
mod-search Orchid	2.1.0-SNAPSHOT.108	3	8	400	2592	2480	1024	1440
mod-inventory-storage Orchid	26.1.0-SNAPSHOT.644	3	2	1024	1952	2208	512	1440

Modules memory and CPU parameters: 5-6 tests tests #5,6

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-search Poppy	3.0.0-SNAPSHOT.156	13	8	2048	2592	2480	512	1440
mod-inventory-storage Poppy	26.1.0-SNAPSHOT.685	11	2	2048	4096	3690	512	3076

...

Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses
Reindex on the new Poppy environment, get the results for indexing time, index size
Compare with results of https://issuesfolio-org.folioatlassian.orgnet/browse/PERF-430

Versions Compared

Old Version 11

New Version Current

Key

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Memory Usage

Poppy

Memory Usage

Poppy

Orchid

CPU Utilization

Poppy

Orchid

CPU Utilization

Poppy

Orchid

RDS CPU Utilization

Poppy

Open Search KPIs

Appendix

Infrastructure

Page Comparison

Versions Compared

Old Version 11

New Version Current

Key

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Memory Usage

Poppy

Memory Usage

Poppy

Orchid

CPU Utilization

Poppy

Orchid

CPU Utilization

Poppy

Orchid

RDS CPU Utilization

Poppy

Open Search KPIs

Appendix

Infrastructure