| Table of Contents |
|---|
...
Test # | Start time | End time | Instances number | Test Conditions reindexing on Ramsons release, non-consortium environment | Duration * | Notes | Objective |
1 | 2024-10-17T12:41:14 | 2024-10-17T14:06:46 | 10,099,620 | Sequential: fs07000001 | 1 hour 25 min |
| We tested mod-search version 4.0.0-SNAPSHOT.281 with the baseline configuration, and it delivered very good results. However, the database size is quite large (using db.r6g.8xlarge), so we plan to experiment with other types of database instances to explore potential optimizations. |
2 | 2024-10-17T14:35:59 | 2024-10-17T19:49:26 | 27,957,839 | Sequential: fs09000000 | 5 hours 14 min | ||
3 | 2024-10-17T19:58:07 | 2024-10-17T20:12:24 | 1,210,000 | Sequential: fs07000002 | 14 min | ||
4 | 2024-10-17T20:21:23 | 2024-10-17T22:46:34 | In parallel: 3 tenants | All tenants reindex FAILED in | |||
5 | 2024-10-16T14:38:08 | 2024-10-16T15:53:37 | 10,099,620 | Sequential: fs07000001 | 1 hour 15 min |
| We tested mod-search version 4.0.0-SNAPSHOT.281 with the following increased memory parameters to assess potential performance improvements:
However, the results showed no significant improvement in performance, indicating that increasing the memory parameters is not beneficial. |
6 | 2024-10-16T16:11:53 | 2024-10-16T21:08:00 | 27,957,839 | Sequential: fs09000000 | 4 hours 57 min | ||
7 | 2024-10-17T06:20:22 | 2024-10-17T06:34:04 | 1,210,000 | Sequential: fs07000002 | 14 min | ||
8 | 2024-10-17T06:40:22 | 2024-10-17T09:12:00 | In parallel: 3 tenants | reindex FAILED for 1 tenant | |||
9 | 2024-10-15T10:16:50 | 2024-10-15T12:04:51 | 10,099,620 | Sequential: fs07000001 | 1 hour 48 min |
| For this version, we encountered out-of-memory (OOM) issues with mod-search and attempted to resolve them by increasing the memory parameters:
However, these adjustments did not resolve the issue, so we updated mod-search to version 4.0.0-SNAPSHOT.281. |
10 | 2024-10-16T08:01:14 | 2024-10-16T12:32:19 | 10,099,620 | Sequential: fs07000001 | 4 hours 31 min |
| This test was performed to check if it is possible to reindex with regular open search size without scaling it. |
11 | 2024-10-21T09:49:10 | 2024-10-21T12:14:38 | 10,099,620 | Sequential: fs07000001 | 2 hours 25 min |
| The baseline database size, using the db.r6g.8xlarge instance, is quite large. A performance test was conducted to assess whether this size is optimal. The results indicate that it would be more efficient to use smaller instances, such as db.r6g.4xlarge or db.r6g.2xlarge, instead. |
12 | 2024-10-21T16:36:18 | 2024-10-21T18:26:15 | 10,099,620 | Sequential: fs07000001 | 1 hour 50 min |
| |
13 | 2024-10-22T09:16:23 | 2024-10-22T10:38:20 | 10,099,620 | Sequential: fs07000001 | 1 hour 22 minutes |
|
...
Database use the same average amount of connections
...
Open Search metrics
CPU utilization percentage for all data nodes (Average).
...
Memory usage percentage for all data nodes (Average).
...
Test #11 2 instances of db.r6g.xlarge database: writer and reader instances.
...
Infrastructure
PTF-environment rcp1
10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zonesApache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
OpenSearch ptf-reindex-testcluster
OpenSearch version 2.13;
Data nodes
Availability Zone(s) - 2-AZ without standby
Instance type - r6g.4xlarge.search
Number of nodes - 4
EBS volume size (GiB) - 300
Provisioned IOPS - 3000IOPS
Provisioned Throughput (MiB/s) - 250 MiB/s
Dedicated master nodes
Enabled - No
...