Overview

The document's purpose is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing. PERF-984 - Getting issue details... STATUS

Test Summary

Recommendations & Jiras

Test Runs /Results

Test #

Total Instances number

Test Conditions

reindexing on Poppy release, consortium environment

Duration

Notes

1

2024-10-22T13:02:35

2024-10-22T16:06:18

In parallel: all tenants

3 hours 4 minutes

Indexing size

All the data from the table below were captured after test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

[
{
"entityType":"HOLDINGS",
"status":"MERGE_COMPLETED",
"totalMergeRanges":26246,
"processedMergeRanges":26246,
"totalUploadRanges":0,
"processedUploadRanges":0,
"startTimeMerge":"2024-10-22T13:02:36.049Z",
"endTimeMerge":"2024-10-22T14:12:31.965Z"
},
{
"entityType":"ITEM",
"status":"MERGE_COMPLETED",
"totalMergeRanges":31369,
"processedMergeRanges":31369,
"totalUploadRanges":0,
"processedUploadRanges":0,
"startTimeMerge":"2024-10-22T13:02:35.944Z",
"endTimeMerge":"2024-10-22T14:06:32.674Z"
},
{
"entityType":"SUBJECT",
"status":"UPLOAD_COMPLETED",
"totalMergeRanges":0,
"processedMergeRanges":0,
"totalUploadRanges":4095,
"processedUploadRanges":4095,
"startTimeUpload":"2024-10-22T14:12:33.759Z",
"endTimeUpload":"2024-10-22T15:11:58.204Z"
},
{
"entityType":"INSTANCE",
"status":"UPLOAD_COMPLETED",
"totalMergeRanges":0,
"processedMergeRanges":0,
"totalUploadRanges":12559,
"processedUploadRanges":12569,
"startTimeUpload":"2024-10-22T14:12:35.052Z",
"endTimeUpload":"2024-10-22T16:06:18.415Z"
},
{
"entityType":"CONTRIBUTOR",
"status":"UPLOAD_COMPLETED",
"totalMergeRanges":0,
"processedMergeRanges":0,
"totalUploadRanges":4095,
"processedUploadRanges":4095,
"startTimeUpload":"2024-10-22T14:12:34.509Z",
"endTimeUpload":"2024-10-22T15:12:19.081Z"
},
{
"entityType":"CLASSIFICATION",
"status":"UPLOAD_COMPLETED",
"totalMergeRanges":0,
"processedMergeRanges":0,
"totalUploadRanges":4095,
"processedUploadRanges":4095,
"startTimeUpload":"2024-10-22T14:12:35.088Z",
"endTimeUpload":"2024-10-22T15:26:05.314Z"
}
]

Resource utilization

Service CPU Utilization

Memory Utilization

Instance CPU Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Subrange of reindexing process from 13:02 - 16:06UTC. A Graph was added to see detailed behavior aggregated on the graph above.

CPU utilization percentage for all data nodes

Memory usage percentage for all data nodes

Average

Maximum

Appendix

Infrastructure

PTF-environment rcon

9 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch ptf-reindex-test cluster
- OpenSearch version 2.13
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

Data structure

Module versions

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcon-pvt

mod-search	2	mod-search:3.2.0	8	2592	2480	2048	1440	512	1024	FALSE
mod-authtoken	1	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512	FALSE
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512	FALSE
mod-users	1	mod-users:19.3.1	2	1024	896	128	768	88	128	FALSE
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512	FALSE

All modules and versions

Methodology/Approach

Use consortium cluster for testing (rcon in our case).
Configure the environment according to Infrastructure parameters and requirements that are in the ticket PERF-889 - Getting issue details... STATUS
Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/full without any parameters on central tenant. For all other tenants in consortium cluster reindex will be performed automatically.
Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status

ECS mod-search: Test Reindexing full (Ramsons)