| Table of Contents |
|---|
Overview
The document's purpose of the document is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing.
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
...
mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search
use the db.r6g.4xlarge or db.r6g.2xlarge database instance
Test Runs /Results
Test # | Start time | End time | Instances number | Test Conditions reindexing on Ramsons release, non-consortium environment | Duration | *Notes | Objective | |||
1 | 2024-10-17T12:41:14 | 2024-10-17T14:06:46 | 10,099,620 | Sequential: fs07000001 | 1 hour 25 min |
| 2 | 2024-10-17T14:35:59 | 2024-10-17T19:49:26 | 27,957,We tested mod-search version 4.0.0-SNAPSHOT.281 with the baseline configuration, and it delivered very good results. However, the database size is quite large (using db.r6g.8xlarge), so we plan to experiment with other types of database instances to explore potential optimizations. |
2 | 2024-10-17T14:35:59 | 2024-10-17T19:49:26 | 27,957,839 | Sequential: fs09000000 | 5 hours 14 min | |||||
3 | 2024-10-17T19:58:07 | 2024-10-17T20:12:24 | 1,210,000 | Sequential: fs07000002 | 14 min | |||||
4 | 2024-10-17T20:21:23 | 2024-10-17T22:46:34 | In parallel: 3 tenants | All tenants reindex FAILED in | ||||||
5 | 2024-10-16T14:38:08 | 2024-10-16T15:53:37 | 10,099,620 | Sequential: fs07000001 | 1 hour 15 min |
| 6 | 2024-10-16T16:11:53 | 2024-10-16T21:08:We tested mod-search version 4.0.0-SNAPSHOT.281 with the following increased memory parameters to assess potential performance improvements:
However, the results showed no significant improvement in performance, indicating that increasing the memory parameters is not beneficial. | |
6 | 2024-10-16T16:11:53 | 2024-10-16T21:08:00 | 27,957,839 | Sequential: fs09000000 | 4 hours 57 min | |||||
7 | 2024-10-17T06:20:22 | 2024-10-17T06:34:04 | 1,210,000 | Sequential: fs07000002 | 14 min | |||||
8 | 2024-10-17T06:40:22 | 2024-10-17T09:12:00 | In parallel: 3 tenants | reindex FAILED for 1 tenant | ||||||
9 | 2024-10-15T10:16:50 | 2024-10-15T12:04:51 | 10,099,620 | Sequential: fs07000001 | 1 hour 48 min |
| For this version, we encountered out-of-memory (OOM) issues with mod-search and attempted to resolve them by increasing the memory parameters:
However, these adjustments did not resolve the issue, so we updated mod-search to version 4.0.0-SNAPSHOT.281. | |||
10 | 2024-10-16T08:01:14 | 2024-10-16T12:32:19 | 10,099,620 | Sequential: fs07000001 | 4 hours 31 min Data nodes instance scaled down to r6g.large.search |
| This test was performed to check if it is possible to reindex with regular open search size without scaling it. | |||
11 | 2024-10-21T09:49:10 | 2024-10-21T12:14:38 | 10,099,620 | Sequential: fs07000001 | 2 hours 25 min |
| The baseline database size, using the db.r6g.8xlarge instance, is quite large. A performance test was conducted to assess whether this size is optimal. The results indicate that it would be more efficient to use smaller instances, such as db.r6g.4xlarge or db.r6g.2xlarge, instead. | |||
12 | 2024-10-21T16:36:18 | 2024-10-21T18:26:15 | 10,099,620 | Sequential: fs07000001 | 1 hour 50 min |
| ||||
13 | 2024-10-22T09:16:23 | 2024-10-22T10:38:20 | 10,099,620 | Sequential: fs07000001 | 1 hour 22 minutes |
|
...
Database use the same average amount of connections
...
Open Search metrics
CPU utilization percentage for all data nodes (Average).
...
Memory usage percentage for all data nodes (Average).
...
Test #11 2 instances of db.r6g.xlarge database: writer and reader instances.
...
Infrastructure
PTF-environment rcp1
10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zonesApache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
OpenSearch ptf-reindex-testcluster
OpenSearch version 2.13;
Data nodes
Availability Zone(s) - 2-AZ without standby
Instance type - r6g.4xlarge.search
Number of nodes - 4
EBS volume size (GiB) - 300
Provisioned IOPS - 3000IOPS
Provisioned Throughput (MiB/s) - 250 MiB/s
Dedicated master nodes
Enabled - No
...
Use a non-consortium cluster for testing (rcp1 in our case).
Configure the environment in accordance with Infrastructure parameters and requirements that are in the ticket
Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-1003 Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status
Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/fullwithout any parameters
Script on the http://github.com/folio-org/perf-testing/mod-search