Overview

The purpose of the document is to compare results of reindexing of instances on Orchid and Poppy releases.

Recommendations & Jiras

Original ticket to test: PERF-637. Compare with test results of PERF-430.
Additional info ElasticSearch Reindex Performance Recommendations
To solve the issue with reindexing RANCHER-1032

Test Summary

Comparing results with PERF-430 reindexing on Poppy release is faster than on Orchid. 9h 47 min against 11h 10 min.

Incorrect behaviour for reindexing was observed during tests #3,4. It was unexpectedly fast (just 1 hour). Jira ticket created RANCHER-1032

After problem was fixed additional 2th tests were carried out. So reindexing results during tests #5,6 are consistent on Poppy release and took 9 hours 46 min and 9 hours 38 min.

It was observed that after 2 hours of instance reindexing all instances are indexed and searchable from UI. The rest of time It goes with ocp2_contributor_fs09000000 indexing.

Average memory usage for mod-search on Orchid - 36% against 47% on Poppy, and for mod-inventory-storage it decreased almost twice on Orchid - 50% against 26% on Poppy).

Average CPU utilization decreased significantly on Poppy (for mod-search on Orchid - 75% against 20% on Poppy, for mod-inventory-storage on Orchid - 50% against 15% on Poppy).

Max. Indexing latency changed from 19 ms on Orchid to 60 ms on Poppy.

Test Runs /Results

Test #	Test Conditions	Duration	Notes
	Reindexing on Orchid release	11 hr 10 min	mod-search task count =8 open search instance scaled up to r6g.4xlarge.search
2. 2023_10_01 10:18 - 20:05 UTC	Reindexing on Poppy release	9 hours 47 min	mod-search task count =8 mod-inventory-storage task count = 2 mod-okapi task count = 3 open search instance scaled up to r6g.4xlarge.search without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch
3. 2023-10-03 10:33 - 11:33 UTC		1 hour Failed
4. 2023-10-03 12:08 - 13:05 UTC		57 min Failed
5. 2023-10-09 17:41 - 03:27 UTC		9 hours 46 min
6. 2023-10-10 06:05 - 15:43 UTC		9 hours 38 min

Indexing size

Test #2

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	ocp2_authority_fs09000000	XE9S-Me_StyzkP3XQqPqBA	4	0	556239	0	233.1mb	233.1mb
green	open	ocp2_contributor_fs09000000	N59vfY3YTsOopikzeqU5NA	4	2	4629419	1300484	14.6gb	4.8gb
green	open	ocp2_instance_fs09000000	vCgkqlfuRYyWExbd8o0BoQ	4	2	10733729	2712	122.9gb	40.9gb
green	open	ocp2_instance_subject_fs09000000	l-07qruwRRKpKR2ec8Gylg	4	2	4071905	1176299	19.7gb	6.5gb

Test #3 Indexing first time after a new OS cluster was created

health	status	index	uuid	pri	rep	docs.count	store.size	pri.store.size
green	open	.opensearch-observability	kPAJ8TqaR06AQZFYekHeyA	1	2	0	624b	208b
green	open	ocp2_instance_fs09000000	61c7lHERQGab0nWBS6tU-w	4	2	10733729	124.5gb	41.3gb
green	open	ocp2_instance_subject_fs09000000	T9QYzSdMS-GFcO4skPIQiw	4	2	0	2.4kb	832b
green	open	ocp2_contributor_fs09000000	qPUffL5HQtWNpyScGOA6cA	4	2	0	2.4kb	832b
green	open	.kibana_1	9Q4bvyKCRpiiwcPNcLFs9g	1	2	1	15.5kb	5.1kb

Test #4

health	status	index	uuid	pri	rep	docs.count	store.size	pri.store.size
green	open	.opensearch-observability	kPAJ8TqaR06AQZFYekHeyA	1	2	0	624b	208b
green	open	ocp2_instance_fs09000000	pj6TE-z6SkeewcFbfr98yg	4	2	10733729	124.7gb	42gb
green	open	ocp2_contributor_fs09000000	lVNwo3siRVKvTGg0_YVivQ	4	2	0	2.4kb	832b
green	open	ocp2_instance_subject_fs09000000	LwlykCF9SrWiu4UV1Spmtw	4	2	0	2.4kb	832b
green	open	.kibana_1	9Q4bvyKCRpiiwcPNcLFs9g	1	2	1	15.5kb	5.1kb

Test #5

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	ocp2_instance_fs09000000	RJFH1FFRTLC2f8tAZuTulg	4	2	10733729	0	123.5gb	40.9gb
green	open	ocp2_authority_fs09000000	9QX5G1CkSMWTIaD-U2Re2g	4	0	556239	0	238.6mb	238.6mb
green	open	ocp2_instance_subject_fs09000000	8VxNRdHjRYOC-fB-G6B4ow	4	2	4076687	736676	18.1gb	6gb
green	open	ocp2_contributor_fs09000000	6pmK601zR3eFwZ02PR0SCg	4	2	4636652	708574	12.6gb	3.8gb

Test #6

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	ocp2_instance_fs09000000	l-8gxUPaRHu9Rb77krV2Pw	4	2	10733729	0	124gb	41.2gb
green	open	ocp2_authority_fs09000000	9QX5G1CkSMWTIaD-U2Re2g	4	0	556239	0	238.6mb	238.6mb
green	open	ocp2_contributor_fs09000000	oxx-7teWRieeiHmj2mZPqw	4	2	4634992	732218	12.8gb	3.8gb
green	open	ocp2_instance_subject_fs09000000	OaxNET2SRO2nG1oPeTy61w	4	2	4076647	472411	18.6gb	6.5gb

Memory Usage

Poppy

Test #2

mod-search max. 74% during an hour, 64% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

Test #3,4

Test #5

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 47%.

mod-inventory-storage avr. 26%

Test #6

mod-search max. 68% during an hour, 60% - during second hour and the rest 7 hours with average 45%.

mod-inventory-storage avr. 26%

Orchid

mod-search avr. 36%

mod-inventory-storage avr. 50%

Memory consumption

Test #	Module	Initial spike	avg. usage
2. Poppy	mod-search	74%	47%
2. Poppy	mod-inventory-storage	26%	26%
5. Poppy	mod-search	68%	47%
5. Poppy	mod-inventory-storage	24%	22%
6. Poppy	mod-search	60%	45%
6. Poppy	mod-inventory-storage	24%	21%
Orchid	mod-search	47%	36%
Orchid	mod-inventory-storage	50%	50%

CPU Utilization

Poppy

Test #2

After 2 hour of reindexing utilization of CPU resources decreased to minimum. 1-2% for mod-inventory-storage and less than 1 % for mod-search

Test #3,4

Test #5

Test #6

CPU utilization

Test #	Poppy release	Initial spike	avg. usage
2	mod-search	32%	20% - first 2 hours. Close to 1% after 2 hours of indexing
2	mod-inventory-storage	49%	15% - first 2 hours. Close to 1% after 2 hours of indexing
5	mod-search	30%	20% - first 2 hours. Close to 1% after 2 hours of indexing
5	mod-inventory-storage	50%	15% - first 2 hours. Close to 1% after 2 hours of indexing
6	mod-search	30%	20% - first 2 hours. Close to 1% after 2 hours of indexing
6	mod-inventory-storage	54%	15% - first 2 hours. Close to 1% after 2 hours of indexing
Orchid	mod-search	200%	75% - first 2 hours. Close to 1% after 2 hours of indexing
Orchid	mod-inventory-storage	100%	50% - first 2 hours. Close to 1% after 2 hours of indexing

Orchid

RDS CPU Utilization

Poppy

Test #2

Utilization max. 63%, avr. 22% during a second hour, after 2 hours of indexing DB utilization decreased to the level as before test start - 6%

Test #3,4

Test #5

Test #6

Spikes after 8:00 can be ignored. DB queries were performed for other ticket purposes this time.

Orchid

Open Search KPIs

Indexing rate Poppy

Test #2

Duration: 9 hours 47 min

Test #3,4

Test #5

Test #6

Indexing rate Orchid

Duration: 11 hours 10 min

Indexing latency Poppy

Test #2

After 7 hours of indexing latency grew from 35 ms to 55 ms during 1 minute

Max: 79.2 ms

Avr.: 39.6 ms

Test #3,4

Test #5

Test #6

Indexing latency Orchid

Appendix

Infrastructure

Tests #2,3,4

PTF -environment ocp2

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 310 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Tests #5,6

PTF -environment ocp2

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6g.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Number of instances

ocp2 -10M

Kafka partitioning

	Orchid	Poppy
inventory.item	50	50
inventory.instance	50	50
inventory.holdings-record	50	50
inventory.bound-with	50	50
inventory.authorit	50	50
search.instance-contributor	50	50

Modules memory and CPU parameters: tests #2,3,4

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-search Poppy	3.0.0-SNAPSHOT.151	6	8	2048	2592	2480	512	1440
mod-inventory-storage Poppy	26.1.0-SNAPSHOT.685	11	2	2048	4096	3690	512	3076
mod-search Orchid	2.1.0-SNAPSHOT.108	3	8	400	2592	2480	1024	1440
mod-inventory-storage Orchid	26.1.0-SNAPSHOT.644	3	2	1024	1952	2208	512	1440

Modules memory and CPU parameters: tests #5,6

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-search Poppy	3.0.0-SNAPSHOT.156	13	8	2048	2592	2480	512	1440
mod-inventory-storage Poppy	26.1.0-SNAPSHOT.685	11	2	2048	4096	3690	512	3076

Methodology/Approach

Use PTF's "Bugfest" Poppy cluster, which has 10M records, to test with (2 times)
Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses
Reindex on the new Poppy environment, get the results for indexing time, index size
Compare with results of https://folio-org.atlassian.net/browse/PERF-430

Browser not supported