/
Non-ECS mod-search: Test Reindexing full (Ramsons)

Non-ECS mod-search: Test Reindexing full (Ramsons)

Owned by Olga Kondratenko
Last updated: Apr 15, 2025 by Roman_Fedynyshyn
14 min read

Overview

The document's purpose is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing. PERF-1003: [Ramsons] [Standalone] Reindex for mod-search (reindex improvements)Closed

Test Summary

  • Reindex could be done in 1 hour 25 minutes (db.r6g.8xlarge) for 10 million instances. It is 6.8 times faster than the Poppy release with db.r6g.xlarge. Duration 2 hours 25 minutes with the db.r6g.xlarge (same database size) and it is 4 times faster compared to Poppy.

  • It is possible to run reindex with the small-size database (xlarge). duration -- hours and we have 10 mln records

  • It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will fail.

  • Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage. For all other services CPU did not exceed 20%.

  • Memory utilization was stable and no memory leaks or OOM issues were observed.

  • RDS CPU utilization was about 90% for the database db.r6g.xlarge and up to 35% for db.r6g.8xlarge.

  • A larger database instance type typically results in faster reindexing times. However, for the db.r6g.8xlarge and db.r6g.4xlarge, the reindexing duration is nearly identical. Therefore, for 10 million instance records reindex, it's more efficient to use the db.r6g.4xlarge or db.r6g.2xlarge database instance.

Recommendations & Jiras

It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will be failed. MSEARCH-868: Unable to run reindex for 3 tenants in parallel Open

Multiple deadlocks in the database were observed at the start of reindex.

It is possible to run reindex on the small-size database (xlarge, 2 instances). duration 2 hours 25 minutes and we have 10 mln records

  • mod-search:

  1. task count = 4

  2. Mem Hard Limit = 2592

  3. Mem Soft Limit = 2480

  4. Xmx = -XX:MaxRAMPercentage=85.0

    "name": "JAVA_OPTS", "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-search.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Dserver.port=8082 -XX:MaxRAMPercentage=85.0"

  • mod-inventory-storage task count = 4

  • open search Data nodes instance scaled up to r6g.4xlarge.search

  • use the db.r6g.4xlarge or db.r6g.2xlarge database instance

Test Runs /Results

Test #

Start time

End time

Instances number

Test Conditions

reindexing on Ramsons release, non-consortium environment

Duration

Notes



Objective

1

2024-10-17T12:41:14

2024-10-17T14:06:46

10,099,620

Sequential: fs07000001

1 hour 25 min

  • mod-search:

  1. task count = 4

  2. Mem Hard Limit = 2592

  3. Mem Soft Limit = 2480

  4. Xmx = -XX:MaxRAMPercentage=85.0

  • mod-inventory-storage task count = 4

  • open search Data nodes instance scaled up to r6g.4xlarge.search

We tested mod-search version 4.0.0-SNAPSHOT.281 with the baseline configuration, and it delivered very good results. However, the database size is quite large (using db.r6g.8xlarge), so we plan to experiment with other types of database instances to explore potential optimizations.

2

2024-10-17T14:35:59

2024-10-17T19:49:26

27,957,839

Sequential: fs09000000

5 hours 14 min

3

2024-10-17T19:58:07

2024-10-17T20:12:24

1,210,000

Sequential: fs07000002

14 min

4

2024-10-17T20:21:23

2024-10-17T22:46:34

 

In parallel: 3 tenants

All tenants reindex FAILED in
2 hours 25 min "MERGE_FAILED"

5

2024-10-16T14:38:08

2024-10-16T15:53:37

10,099,620

Sequential: fs07000001

1 hour 15 min

  • mod-search:

  1. task count = 4

  2. Mem Hard Limit = 4592

  3. Mem Soft Limit = 4480

  4. Xmx = -XX:MaxRAMPercentage=85.0

  • mod-inventory-storage task count = 4

  • open search Data nodes instance scaled up to r6g.4xlarge.search

We tested mod-search version 4.0.0-SNAPSHOT.281 with the following increased memory parameters to assess potential performance improvements:

  • Mem Hard Limit: 4592 MB

  • Mem Soft Limit: 4480 MB

However, the results showed no significant improvement in performance, indicating that increasing the memory parameters is not beneficial.

6

2024-10-16T16:11:53

2024-10-16T21:08:00

27,957,839

Sequential: fs09000000

4 hours 57 min

7

2024-10-17T06:20:22

2024-10-17T06:34:04

1,210,000

Sequential: fs07000002

14 min

8

2024-10-17T06:40:22

2024-10-17T09:12:00

 

In parallel: 3 tenants

reindex FAILED for 1 tenant [tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]

9

2024-10-15T10:16:50

2024-10-15T12:04:51

10,099,620

Sequential: fs07000001

1 hour 48 min

  • mod-search:4.0.0-SNAPSHOT.278

For this version, we encountered out-of-memory (OOM) issues with mod-search and attempted to resolve them by increasing the memory parameters:

  • Mem Hard Limit: 4592 MB

  • Mem Soft Limit: 4480 MB

However, these adjustments did not resolve the issue, so we updated mod-search to version 4.0.0-SNAPSHOT.281.

10

2024-10-16T08:01:14

2024-10-16T12:32:19

10,099,620

Sequential: fs07000001

4 hours 31 min

  • Data nodes instance scaled down to r6g.large.search

This test was performed to check if it is possible to reindex with regular open search size without scaling it.
It is possible to reindex but it will take longer to reindex (approx 4 times longer for r6g.large.search compared to r6g.4xlarge.search)

11

2024-10-21T09:49:10

2024-10-21T12:14:38

10,099,620

Sequential: fs07000001

2 hours 25 min

  • 2 instances of database db.r6g.xlarge (reader and writer)

The baseline database size, using the db.r6g.8xlarge instance, is quite large. A performance test was conducted to assess whether this size is optimal. The results indicate that it would be more efficient to use smaller instances, such as db.r6g.4xlarge or db.r6g.2xlarge, instead.

12

2024-10-21T16:36:18

2024-10-21T18:26:15

10,099,620

Sequential: fs07000001

1 hour 50 min

  • 1 instance of database db.r6g.2xlarge

13

2024-10-22T09:16:23

2024-10-22T10:38:20

10,099,620

Sequential: fs07000001

1 hour 22 minutes

  • 1 instance of database db.r6g.4xlarge

Indexing size

All the data from the tables below were captured after each test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

Test #1
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.778Z","endTimeMerge":"2024-10-17T13:08:31.434Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.839Z","endTimeMerge":"2024-10-17T13:13:52.870Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.544Z","endTimeUpload":"2024-10-17T13:50:47.667Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.941Z","endTimeUpload":"2024-10-17T14:02:34.736Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-17T13:14:08.598Z","endTimeUpload":"2024-10-17T14:06:09.460Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:09.193Z","endTimeUpload":"2024-10-17T14:06:46.360Z"}]
Test #2
[{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-17T16:57:51.729Z","endTimeUpload":"2024-10-17T19:49:26.148Z"}, {"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:35:59.909Z","endTimeMerge":"2024-10-17T16:42:19.528Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:36:00.032Z","endTimeMerge":"2024-10-17T16:57:34.560Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.682Z","endTimeUpload":"2024-10-17T17:46:13.105Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.850Z","endTimeUpload":"2024-10-17T18:04:25.328Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:52.021Z","endTimeUpload":"2024-10-17T19:03:31.441Z"}]
Test #3
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.803Z","endTimeMerge":"2024-10-17T20:03:45.240Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.813Z","endTimeMerge":"2024-10-17T20:04:19.431Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T20:04:20.952Z","endTimeUpload":"2024-10-17T20:12:07.684Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.959Z","endTimeUpload":"2024-10-17T20:12:13.154Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.986Z","endTimeUpload":"2024-10-17T20:12:17.030Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:21.037Z","endTimeUpload":"2024-10-17T20:12:24.156Z"}]
Test #4
[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":57956,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.508Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":55778,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.638Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":55660,"processedMergeRanges":19214,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.324Z","endTimeMerge":"2024-10-17T22:46:34.030Z"}] [{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2970,"processedMergeRanges":2065,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.717Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":20202,"processedMergeRanges":17676,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.700Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":21045,"processedMergeRanges":13092,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.762Z","endTimeMerge":"2024-10-17T20:53:24.774Z"}] [{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":2423,"processedMergeRanges":2422,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.851Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.877Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.900Z","endTimeMerge":"2024-10-17T20:23:48.044Z"}]
Test #5
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.913Z","endTimeMerge":"2024-10-16T15:04:31.431Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.957Z","endTimeMerge":"2024-10-16T15:07:12.367Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.170Z","endTimeUpload":"2024-10-16T15:40:17.836Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.576Z","endTimeUpload":"2024-10-16T15:51:05.755Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.795Z","endTimeUpload":"2024-10-16T15:53:35.729Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T15:07:26.195Z","endTimeUpload":"2024-10-16T15:53:37.124Z"}]
Test #6
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.185Z","endTimeMerge":"2024-10-16T18:15:03.509Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.305Z","endTimeMerge":"2024-10-16T18:30:07.978Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.240Z","endTimeUpload":"2024-10-16T19:15:39.436Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.398Z","endTimeUpload":"2024-10-16T19:30:23.501Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.522Z","endTimeUpload":"2024-10-16T20:32:58.360Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-16T18:30:22.281Z","endTimeUpload":"2024-10-16T21:08:00.747Z"}]
Test #7
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.898Z","endTimeMerge":"2024-10-17T06:25:56.268Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.905Z","endTimeMerge":"2024-10-17T06:26:35.476Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T06:26:37.038Z","endTimeUpload":"2024-10-17T06:33:52.000Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.241Z","endTimeUpload":"2024-10-17T06:33:56.853Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.481Z","endTimeUpload":"2024-10-17T06:33:57.691Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.807Z","endTimeUpload":"2024-10-17T06:34:04.078Z"}]
Test #9
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.469Z","endTimeMerge":"2024-10-15T10:50:57.419Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.534Z","endTimeMerge":"2024-10-15T10:53:48.693Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.365Z","endTimeUpload":"2024-10-15T11:50:19.948Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.797Z","endTimeUpload":"2024-10-15T12:00:12.641Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-15T10:54:04.383Z","endTimeUpload":"2024-10-15T12:04:14.378Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:05.024Z","endTimeUpload":"2024-10-15T12:04:51.641Z"}]
Test #10
[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.876Z","endTimeMerge":"2024-10-16T08:28:09.875Z"}, {"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.935Z","endTimeMerge":"2024-10-16T08:33:38.238Z"}, {"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:42.418Z","endTimeUpload":"2024-10-16T11:52:29.684Z"}, {"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:43.531Z","endTimeUpload":"2024-10-16T12:30:17.561Z"}, {"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T08:33:42.450Z","endTimeUpload":"2024-10-16T12:31:53.638Z"}, {"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:44.015Z","endTimeUpload":"2024-10-16T12:32:19.468Z"}]

Compared to Poppy

Compared to the test with instances of 10 mln records size.
Difference in configuration:

Configuration

Ramsons

Poppy

Configuration

Ramsons

Poppy

Database

1 instance of db.r6g.8xlarge database instances, writer

2 instances of db.r6g.xlarge database instances, one reader, and one writer

mod-search version

4.0.0-SNAPSHOT.281

3.0.0-SNAPSHOT.151

mod-search task count

4

8

Data node instances count (r6g.4xlarge.search)

4

6

Dedicated master nodes

No

3 of r6g.large.search instances

Duration comparison:

 

Ramsons

Poppy

Delta absolut

Delta

 

Ramsons

Poppy

Delta absolut

Delta

Ramsons database instance type r6g.8xlarge according to requirements PERF-1003: [Ramsons] [Standalone] Reindex for mod-search (reindex improvements)Closed

1 hour 25 min

9 hours 38 min

8 hours 13 min

6.8 times

Ramsons: 2 instances of database instance type r6g.xlarge (the same as for Poppy testing)

2 hours 25 min

9 hours 38 min

7 hours 13 min

4 times

Reindex duration and database size correlation:

A larger database instance type typically results in faster reindexing times. However, for the db.r6g.8xlarge and db.r6g.4xlarge, the reindexing duration is nearly identical. Therefore, for 10 million instance records reindex, it's more efficient to use the db.r6g.4xlarge or db.r6g.2xlarge database instance.

Database size

Duration

Database size

Duration

2 instances of database database db.r6g.xlarge

2 hours 25 min

database db.r6g.2xlarge

1 hour 50 min

database db.r6g.4xlarge

1 hour 22 min

database db.r6g.8xlarge

1 hour 25 min

Resource utilization

Service CPU Utilization

Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage, for all other services CPU did not exceed 20%.

chrome_KdBdej7vNy.png

 

chrome_EuziC30F8G.png

 

 

chrome_MFUhZrpI7N.png

 

Instance CPU Utilization

chrome_o5pH7EPy9Q.png


Memory Utilization

 

chrome_L6KXJImq2X.png

 

chrome_Fk2eN5yM7X.png

DB CPU Utilization

Average database CPU utilization was up to 35%

chrome_P3guvp8nk9.png

 

DB Connections

Database use the same average amount of connections

chrome_fcF34bIjby.png

Open Search metrics

CPU utilization percentage for all data nodes (Average).

chrome_4bWAtGZ5AV.png

 

Memory usage percentage for all data nodes (Average).

chrome_6YwE1SP0iQ.png
chrome_rFZPbdNytz.png

 

chrome_tuP3wGC9cE.png

Test #11 2 instances of db.r6g.xlarge database: writer and reader instances.

 

 

chrome_W9BKHeRa8W.png

Appendix

Infrastructure

PTF-environment rcp1

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 1 instance of db.r6g.8xlarge database, writer instance.

  • MSK - fse-tenant
    4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearch ptf-reindex-test cluster

    • OpenSearch version 2.13;

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.4xlarge.search

      • Number of nodes - 4

      • EBS volume size (GiB) - 300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

    • Dedicated master nodes
      Enabled - No

Data structure

Module
Cluster Resources -

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-search

10

mod-search:4.0.0-SNAPSHOT.281

4

2592

2480

2048

MaxRAMPercentage=85.0

512

1024

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.2.0-SNAPSHOT.773

4

4096

3690

2048

3076

512

1024

mod-inventory

1

mod-inventory:20.3.0-SNAPSHOT.546

2

2880

2592

1024

1814

384

512

mod-users

1

mod-users:19.3.2-SNAPSHOT.696

2

1024

896

0

768

88

128

nginx-okapi

1

nginx-okapi:2023.06.14

2

1024

896

0

0

0

0

okapi-b

1

okapi:5.3.0

3

1684

1440

1024

922

384

512

pub-okapi

1

pub-okapi:2023.06.14

2

1024

896

0

768

0

0

Module versions

Cluster Resources - rcp1

 

 

 

 

 

 

 

 

 

Module

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-remote-storage

1

mod-remote-storage:3.2.1-SNAPSHOT.171

2

4920

4472

1024

3960

512

512

mod-ncip

1

mod-ncip:1.14.6-SNAPSHOT.233

2

1024

896

0

768

88

128

mod-finance-storage

1

mod-finance-storage:8.7.0-SNAPSHOT.181

2

1024

896

1024

700

88

128

mod-agreements

1

mod-agreements:7.1.0-SNAPSHOT.237

2

1592

1488

0

0

0

0

mod-ebsconet

1

mod-ebsconet:2.3.0-SNAPSHOT.80

2

1248

1024

128

700

128

256

mod-organizations

1

mod-organizations:2.0.0-SNAPSHOT.93

2

1024

896

0

700

88

128

edge-sip2

1

edge-sip2:3.3.0-SNAPSHOT.264

2

1024

896

0

768

88

128

mod-settings

1

mod-settings:1.0.4-SNAPSHOT.67

2

1024

896

200

768

88

128

mod-serials-management

1

mod-serials-management:1.1.0-SNAPSHOT.46

2

2480

2312

0

1792

384

512

edge-dematic

1

edge-dematic:2.3.0-SNAPSHOT.143

1

1024

896

0

768

88

128

mod-data-import

1

mod-data-import:3.2.0-SNAPSHOT.189

1

2048

1844

256

1292

384

512

mod-search

10

mod-search:4.0.0-SNAPSHOT.281

4

2592

2480

2048

MaxRAMPercentage=85.0

512

1024

mod-inn-reach

1

mod-inn-reach:3.2.1-SNAPSHOT.102

2

3600

3240

1024

2880

512

1024

mod-record-specifications

10

mod-record-specifications:1.0.0-SNAPSHOT.4

2

1024

896

0

768

88

128

mod-tags

1

mod-tags:2.2.1-SNAPSHOT.138

2

1024

896

0

768

88

128

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

edge-courses

1

edge-courses:1.5.0-SNAPSHOT.1160

2

1024

896

0

768

88

128

mod-notify

1

mod-notify:3.2.1-SNAPSHOT.268

2

1024

896

0

768

88

128

mod-inventory-update

1

mod-inventory-update:3.4.2-SNAPSHOT.100

2

1024

896

0

768

88

128

mod-configuration

1

mod-configuration:5.11.0-SNAPSHOT.355

2

1024

896

0

768

88

128

mod-orders-storage

1

mod-orders-storage:13.8.0-SNAPSHOT.243

2

1024

896

512

700

88

128

edge-caiasoft

1

edge-caiasoft:2.3.0-SNAPSHOT.105

2

1024

896

0

768

88

128

mod-login-saml

1

mod-login-saml:2.9.0-SNAPSHOT.291

2

1024

896

0

768

88

128

mod-erm-usage-harvester

1

mod-erm-usage-harvester:5.0.0-SNAPSHOT.133

2

1024

896

0

768

88

128

mod-password-validator

1

mod-password-validator:3.2.2-SNAPSHOT.149

2

1440

1298

0

768

384

512

mod-gobi

1

mod-gobi:2.9.0-SNAPSHOT.259

2

1024

896

0

700

88

128

mod-licenses

1

mod-licenses:6.1.0-SNAPSHOT.163

2

2480

2312

0

1792

384

512

edge-dcb

1

edge-dcb:1.2.0-SNAPSHOT.39

2

1024

896

0

768

88

128

mod-bulk-operations

1

mod-bulk-operations:2.1.0-SNAPSHOT.119

2

3072

2600

1024

1536

384

512

mod-fqm-manager

10

mod-fqm-manager:2.1.0-SNAPSHOT.1197

2

3000

2600

128

2048

384

512

mod-graphql

1

mod-graphql:1.12.1.412

2

1024

896

0

768

88

128

mod-finance

1

mod-finance:5.0.0-SNAPSHOT.174

2

1024

896

0

700

88

128

mod-erm-usage

1

mod-erm-usage:5.0.0-SNAPSHOT.109

2

1024

896

0

768

88

128

mod-batch-print

1

mod-batch-print:1.2.0-SNAPSHOT.15

2

1024

896

0

768

88

128

mod-tlr

2

mod-tlr:1.0.0-SNAPSHOT.8

2

1024

896

0

768

88

128

mod-lists

5

mod-lists:2.1.0-SNAPSHOT.1303

2

1024

896

0

768

88

128

mod-copycat

1

mod-copycat:1.6.1-SNAPSHOT.208

2

1024

512

0

768

88

128

mod-entities-links

4

mod-entities-links:3.1.0-SNAPSHOT.113

2

2592

2480

400

1440

0

1024

mod-permissions

2

mod-permissions:6.6.0-SNAPSHOT.385

2

1684

1544

512

1024

384

512

pub-edge

1

pub-edge:2023.06.14

2

1024

896

0

768

0

0

mod-orders

1

mod-orders:12.9.0-SNAPSHOT.398

2

2048

1440

1024

1024

384

512

edge-patron

1

edge-patron:5.2.0-SNAPSHOT.174

2

1024

896

256

768

88

128

edge-ncip

1

edge-ncip:1.10.2-SNAPSHOT.116

2

1024

896

0

768

88

128

mod-marc-migrations

26

mod-marc-migrations:1.0.0-SNAPSHOT.8

2

1024

896

0

768

88

128

edge-inn-reach

1

edge-inn-reach:3.3.0-SNAPSHOT.69

2

1024

896

0

768

88

128

mod-users-bl

1

mod-users-bl:7.9.0-SNAPSHOT.318

2

1440

1152

512

922

88

128

mod-oa

1

mod-oa:2.1.0-SNAPSHOT.66

2

1024

896

0

768

88

128

mod-invoice

1

mod-invoice:5.9.0-SNAPSHOT.205

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.2.0-SNAPSHOT.773

4

4096

3690

2048

3076

512

1024

mod-user-import

1

mod-user-import:3.9.0-SNAPSHOT.197

2

1024

896

0

768

88

128

mod-sender

1

mod-sender:1.12.1-SNAPSHOT.100

2

1024

896

0

768

88

128

edge-oai-pmh

1

edge-oai-pmh:2.10.0-SNAPSHOT.174

2

1512

1360

1024

1440

384

512

mod-data-export-worker

1

mod-data-export-worker:3.3.0-SNAPSHOT.233

2

3072

2048

1024

2048

384

512

mod-rtac

1

mod-rtac:3.7.0-SNAPSHOT.203

2

1024

896

0

768

88

128

mod-circulation-storage

1

mod-circulation-storage:17.3.0-SNAPSHOT.405

2

2880

2592

1536

1814

384

512

mod-source-record-storage

2

mod-source-record-storage:5.9.0-SNAPSHOT.405

2

5600

5000

2048

3500

384

512

mod-calendar

2

mod-calendar:3.2.0-SNAPSHOT.229

2

1024

896

0

768

88

128

mod-event-config

1

mod-event-config:2.7.2-SNAPSHOT.110

2

1024

896

0

768

88

128

mod-courses

1

mod-courses:1.4.11-SNAPSHOT.155

2

1024

896

0

768

88

128

mod-circulation-item

1

mod-circulation-item:1.0.1-SNAPSHOT.26

2

1024

896

0

0

0

0

mod-inventory

1

mod-inventory:20.3.0-SNAPSHOT.546

2

2880

2592

1024

1814

384

512

mod-email

1

mod-email:1.17.1-SNAPSHOT.128

2

1024

896

0

768

88

128

mod-requests-mediated

10

mod-requests-mediated:1.0.0-SNAPSHOT.4

2

1024

896

0

768

88

128

mod-di-converter-storage

1

mod-di-converter-storage:2.3.0-SNAPSHOT.124

2

1024

896

0

768

88

128

mod-pubsub

1

mod-pubsub:2.15.0-SNAPSHOT.224

2

1536

1440

1024

922

384

512

mod-circulation

1

mod-circulation:24.3.0-SNAPSHOT.681

2

2880

2592

1536

1814

384

512

edge-orders

1

edge-orders:3.1.0-SNAPSHOT.156

2

1024

896

0

768

88

128

edge-rtac

1

edge-rtac:2.7.3-SNAPSHOT.234

2

1024

896

0

768

88

128

mod-template-engine

1

mod-template-engine:1.20.1-SNAPSHOT.118

2

1024

896

0

768

88

128

mod-users

1

mod-users:19.3.2-SNAPSHOT.696

2

1024

896

0

768

88

128

mod-patron-blocks

1

mod-patron-blocks:1.11.0-SNAPSHOT.163

2

1024

896

1024

768

88

128

mod-audit

1

mod-audit:2.9.1-SNAPSHOT.216

2

1024

896

0

768

88

128

edge-fqm

1

edge-fqm:2.1.0-SNAPSHOT.100

2

1024

896

0

768

88

128

mod-source-record-manager

1

mod-source-record-manager:3.9.0-SNAPSHOT.353

2

5600

5000

2048

3500

384

512

nginx-edge

1

nginx-edge:2023.06.14

2

1024

896

0

0

0

0

mod-quick-marc

1

mod-quick-marc:5.2.0-SNAPSHOT.153

1

2288

2176

0

1664

384

512

nginx-okapi

1

nginx-okapi:2023.06.14

2

1024

896

0

0

0

0

okapi-b

1

okapi:5.3.0

3

1684

1440

1024

922

384

512

mod-feesfines

1

mod-feesfines:19.2.0-SNAPSHOT.198

2

1024

896

0

768

88

128

mod-invoice-storage

1

mod-invoice-storage:5.9.0-SNAPSHOT.138

2

1872

1536

1024

1024

384

512

mod-reading-room

2

mod-reading-room:1.0.0-SNAPSHOT.12

2

1024

896

0

768

88

128

mod-service-interaction

1

mod-service-interaction:4.1.0-SNAPSHOT.126

2

2048

1844

256

1290

384

512

mod-dcb

1

mod-dcb:1.1.3-SNAPSHOT.40

2

1024

896

0

768

88

128

mod-data-export

1

mod-data-export:5.1.0-SNAPSHOT.240

1

2048

1524

1024

0

0

0

mod-patron

1

mod-patron:6.2.0-SNAPSHOT.165

2

1024

896

0

768

88

128

mod-oai-pmh

1

mod-oai-pmh:3.14.0-SNAPSHOT.402

2

4096

3690

2048

3076

384

512

edge-connexion

1

edge-connexion:1.4.0-SNAPSHOT.88

2

1024

896

0

768

88

128

mod-notes

1

mod-notes:5.3.0-SNAPSHOT.325

2

1024

896

0

952

384

512

mod-kb-ebsco-java

1

mod-kb-ebsco-java:4.2.0-SNAPSHOT.178

2

1024

896

0

768

88

128

mod-login

1

mod-login:7.12.0-SNAPSHOT.324

2

1440

1298

1024

768

384

512

mod-organizations-storage

1

mod-organizations-storage:4.8.0-SNAPSHOT.131

2

1024

896

0

700

88

128

mod-data-export-spring

1

mod-data-export-spring:3.3.1-SNAPSHOT.167

1

2048

1844

256

1536

384

512

pub-okapi

1

pub-okapi:2023.06.14

2

1024

896

0

768

0

0

mod-eusage-reports

1

mod-eusage-reports:3.0.0-SNAPSHOT.73

2

1024

896

0

768

88

128

Methodology/Approach

 

 

Additional Testing with mod-search:4.0.11

Overview

Retesting of reindex In scope of PERF-1087: [Ramsons] [ECS] Reindex for mod-search (Call Number Browse Refactor)Closed after call number browse refactoring. Goal of retest is to measure performance of mod-search reindex and it affect on DI (Data Import) running on the background.

Whole process of reindex consists of two phases - “Merging“ and “Uploading“, takes place in sequence.

  • “Merging” phase consists of “items merging“ and “holdings merging“ (happening at the same time, concurrently)

  • “Uploading” phase consists of Instance, Subject, Contributor, Classification, Call_number uploads to OS (open search). Happing at the same time, concurrently.

Summary

 

 

 

 

Results

Test #

Merge phase

Upload phase

Comment

Test #

Merge phase

Upload phase

Comment

1

2 hr 56 min

2 hr 11 min

 

2

2 hr 59 min

2 hr 11 min

DI 25K create instances holdings and items + marc BIB on the background

Detailed results analysis/comparison

 

[

  {

    "entityType": "ITEM",

    "status": "MERGE_COMPLETED",

    "totalMergeRanges": 64914,

    "processedMergeRanges": 64914,

    "totalUploadRanges": 0,

    "processedUploadRanges": 0,

    "startTimeMerge": "2025-04-09T17:26:12.292Z",

    "endTimeMerge": "2025-04-09T20:07:57.932Z"

  },

  {

    "entityType": "CONTRIBUTOR",

    "status": "UPLOAD_COMPLETED",

    "totalMergeRanges": 0,

    "processedMergeRanges": 0,

    "totalUploadRanges": 4096,

    "processedUploadRanges": 4096,

    "startTimeUpload": "2025-04-09T20:22:47.378Z",

    "endTimeUpload": "2025-04-09T20:25:23.145Z"

  },

  {

    "entityType": "HOLDINGS",

    "status": "MERGE_FAILED",

    "totalMergeRanges": 62736,

    "processedMergeRanges": 62736,

    "totalUploadRanges": 0,

    "processedUploadRanges": 0,

    "startTimeMerge": "2025-04-09T17:26:12.411Z",

    "endTimeMerge": "2025-04-09T20:22:46.105Z"

  },

  {

    "entityType": "CLASSIFICATION",

    "status": "UPLOAD_COMPLETED",

    "totalMergeRanges": 0,

    "processedMergeRanges": 0,

    "totalUploadRanges": 4096,

    "processedUploadRanges": 4096,

    "startTimeUpload": "2025-04-09T20:22:47.787Z",

    "endTimeUpload": "2025-04-09T20:25:50.078Z"

  },

  {

    "entityType": "CALL_NUMBER",

    "status": "UPLOAD_COMPLETED",

    "totalMergeRanges": 0,

    "processedMergeRanges": 0,

    "totalUploadRanges": 4096,

    "processedUploadRanges": 4096,

    "startTimeUpload": "2025-04-09T20:22:48.103Z",

    "endTimeUpload": "2025-04-09T20:27:00.767Z"

  },

  {

    "entityType": "SUBJECT",

    "status": "UPLOAD_FAILED",

    "totalMergeRanges": 0,

    "processedMergeRanges": 0,

    "totalUploadRanges": 4096,

    "processedUploadRanges": 4096,

    "startTimeUpload": "2025-04-09T20:22:46.623Z",

    "endTimeUpload": "2025-04-09T20:22:48.124Z"

  },

  {

    "entityType": "INSTANCE",

    "status": "UPLOAD_COMPLETED",

    "totalMergeRanges": 0,

    "processedMergeRanges": 0,

    "totalUploadRanges": 31428,

    "processedUploadRanges": 31428,

    "startTimeUpload": "2025-04-09T20:22:56.276Z",

    "endTimeUpload": "2025-04-09T22:33:42.910Z"

  }

]

 

[ { "entityType": "HOLDINGS", "status": "MERGE_COMPLETED", "totalMergeRanges": 62736, "processedMergeRanges": 62736, "totalUploadRanges": 0, "processedUploadRanges": 0, "startTimeMerge": "2025-04-10T06:03:22.457Z", "endTimeMerge": "2025-04-10T09:01:25.038Z" }, { "entityType": "CALL_NUMBER", "status": "UPLOAD_COMPLETED", "totalMergeRanges": 0, "processedMergeRanges": 0, "totalUploadRanges": 4096, "processedUploadRanges": 4096, "startTimeUpload": "2025-04-10T09:01:27.510Z", "endTimeUpload": "2025-04-10T09:05:35.438Z" }, { "entityType": "INSTANCE", "status": "UPLOAD_COMPLETED", "totalMergeRanges": 0, "processedMergeRanges": 0, "totalUploadRanges": 31453, "processedUploadRanges": 31453, "startTimeUpload": "2025-04-10T09:01:43.224Z", "endTimeUpload": "2025-04-10T11:12:33.133Z" }, { "entityType": "CLASSIFICATION", "status": "UPLOAD_COMPLETED", "totalMergeRanges": 0, "processedMergeRanges": 0, "totalUploadRanges": 4096, "processedUploadRanges": 4096, "startTimeUpload": "2025-04-10T09:01:26.521Z", "endTimeUpload": "2025-04-10T09:04:12.736Z" }, { "entityType": "ITEM", "status": "MERGE_COMPLETED", "totalMergeRanges": 64914, "processedMergeRanges": 64914, "totalUploadRanges": 0, "processedUploadRanges": 0, "startTimeMerge": "2025-04-10T06:03:22.296Z", "endTimeMerge": "2025-04-10T08:47:26.298Z" }, { "entityType": "SUBJECT", "status": "UPLOAD_FAILED", "totalMergeRanges": 0, "processedMergeRanges": 0, "totalUploadRanges": 4096, "processedUploadRanges": 4096, "startTimeUpload": "2025-04-10T09:01:25.489Z", "endTimeUpload": "2025-04-10T09:01:35.633Z" }, { "entityType": "CONTRIBUTOR", "status": "UPLOAD_COMPLETED", "totalMergeRanges": 0, "processedMergeRanges": 0, "totalUploadRanges": 4096, "processedUploadRanges": 4096, "startTimeUpload": "2025-04-10T09:01:26.108Z", "endTimeUpload": "2025-04-10T09:03:25.441Z" } ]

Entity

Records (Test 1)

Records (Test 2)

Duration (Test 1)

Duration (Test 2)

Duration Change

Entity

Records (Test 1)

Records (Test 2)

Duration (Test 1)

Duration (Test 2)

Duration Change

ITEM

64,914

64,914

17:26–20:07 (~2h 41m)

06:03–08:47 (~2h 44m)

+3 min

HOLDINGS

62,736

62,736

17:26–20:22 (~2h 56m, ❌ Failed)

06:03–09:01 (~2h 58m, ✅ Success)

+2 min

 

Entity

Ranges (Test 1)

Ranges (Test 2)

Duration (Test 1)

Duration (Test 2)

Duration Change

Entity

Ranges (Test 1)

Ranges (Test 2)

Duration (Test 1)

Duration (Test 2)

Duration Change

INSTANCE

31,428

31,453

20:22–22:33 (~2h 11m)

09:01–11:12 (~2h 11m)

Same

SUBJECT

4,096

4,096

20:22:46–20:22:48 (~2s, ❌)

09:01:25–09:01:35 (~10s, ❌)

Slightly slower

CONTRIBUTOR

4,096

4,096

20:22–20:25 (~2m 36s)

09:01–09:03 (~2m 59s)

+23s

CLASSIFICATION

4,096

4,096

20:22–20:25 (~2m 3s)

09:01–09:04 (~2m 46s)

+43s

CALL_NUMBER

4,096

4,096

20:22–20:27 (~4m 13s)

09:01–09:05 (~4m 8s)

~Same

 

 

Resource utilization

OpenSearch metrics

image-20250415-071436.png
image-20250415-071627.png

 

image-20250415-071842.png

 

image-20250415-072026.png

 

image-20250415-071721.png

 

 

 

Services metrics

 

image-20250414-083527.png

 

 

image-20250414-083607.png

 

DB Metrics

image-20250414-084314.png
image-20250414-085428.png
image-20250414-085459.png

 

image-20250415-073415.png

 

Infrastructure

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

rcon-pvt

mod-search

28

mod-search:4.0.11

8

2592

2480

0

1440

512

1024

mod-inventory-storage

20

mod-inventory-storage:28.0.9

2

4096

3690

0

3076

512

1024

 

PTF-environment rcp1

  • 6 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 1 instance of db.r6g.8xlarge database, writer instance.

  • MSK - fse-tenant
    4 kafka.m7g.xlarge brokers in 2 zones

    • Kafka mode: KRaft

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearch ptf-reindex-test cluster

    • OpenSearch version 2.13

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.2xlarge.search

      • Number of nodes - 4

      • EBS volume size (GiB) - 300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

    • Dedicated master nodes
      Enabled - No

Related content