Table of Contents

Overview

The document's purpose of the document is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-1003

Recommendations & Jiras

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480

Xmx = -XX:MaxRAMPercentage=85.0

Code Block

                    "name": "JAVA_OPTS",
                    "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-search.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Dserver.port=8082 -XX:MaxRAMPercentage=85.0"

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search
1 instance of db.r6g.8xlarge database instances, writer

Test Summary

Test Runs /Results

...

Test #

...

Start time

...

End time

...

Instances number

...

Test Conditions

reindexing on Poppy release, consortium environment

...

Duration *

Notes

...

1

...

2024-10-17T12:41:14

...

Test Summary

Reindex could be done in 1 hour 25 minutes (db.r6g.8xlarge) for 10 million instances. It is 6.8 times faster than the Poppy release with db.r6g.xlarge. Duration 2 hours 25 minutes with the db.r6g.xlarge (same database size) and it is 4 times faster compared to Poppy.
It is possible to run reindex with the small-size database (xlarge). duration -- hours and we have 10 mln records
It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will fail.
Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage. For all other services CPU did not exceed 20%.
Memory utilization was stable and no memory leaks or OOM issues were observed.
RDS CPU utilization was about 90% for the database db.r6g.xlarge and up to 35% for db.r6g.8xlarge.
A larger database instance type typically results in faster reindexing times. However, for the db.r6g.8xlarge and db.r6g.4xlarge, the reindexing duration is nearly identical. Therefore, for 10 million instance records reindex, it's more efficient to use the db.r6g.4xlarge or db.r6g.2xlarge database instance.

Recommendations & Jiras

It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will be failed.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MSEARCH-868

Multiple deadlocks in the database were observed at the start of reindex.

It is possible to run reindex on the small-size database (xlarge, 2 instances). duration 2 hours 25 minutes and we have 10 mln records

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480

Xmx = -XX:MaxRAMPercentage=85.0

Code Block

                    "name": "JAVA_OPTS",
                    "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-search.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Dserver.port=8082 -XX:MaxRAMPercentage=85.0"

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search
use the db.r6g.4xlarge or db.r6g.2xlarge database instance

Test Runs /Results

2task count = 4 = =

Xmx = -XX:MaxRAMPercentage=85.0

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

Test #

Start time

End time

Instances number

Test Conditions

reindexing on Ramsons release, non-consortium environment

Duration

Notes

Objective

1

2024-10-17T12:41:14

2024-10-17T14:06:46

10,099,620

Sequential: fs07000001

1 hour 25 min

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480
Xmx = -XX:MaxRAMPercentage=85.0

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

We tested mod-search version 4.0.0-SNAPSHOT.281 with the baseline configuration, and it delivered very good results. However, the database size is quite large (using db.r6g.8xlarge), so we plan to experiment with other types of database instances to explore potential optimizations.

2

2024-10-17T14:35:59

2024-10-17T19:49:26

27,957,839

Sequential: fs09000000

5 hours 14 min

3

2024-10-17T19:58:07

2024-10-17T20:12:24

1,210,000

Sequential: fs07000002

14 min

4

2024-10-17T20:21:23

2024-10-17T22:46:34

In parallel: 3 tenants

All tenants reindex FAILED in
2 hours 25 min "MERGE_FAILED"

5

2024-10-16T14:38:08

2024-10-16T15:53:37

10,099,620

Sequential: fs07000001

1 hour 25 15 min

mod-search:

task count = 4
Mem Hard Limit = 2592 4592
Mem Soft Limit = 2480 4480
Xmx = -XX:MaxRAMPercentage=85.0

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

to r6g.4xlarge.search

We tested mod-search version 4.0.0-SNAPSHOT.281 with the following increased memory parameters to assess potential performance improvements:

Mem Hard Limit: 4592 MB
Mem Soft Limit: 4480 MB

However, the results showed no significant improvement in performance, indicating that increasing the memory parameters is not beneficial.

6

2024-10-17T1416T16:3511:5953

2024-10-17T1916T21:4908:2600

27,957,839

Sequential: fs09000000

5 4 hours 14 57 min

37

2024-10-17T1917T06:5820:0722

2024-10-17T2017T06:1234:2404

1,210,000

Sequential: fs07000002

14 min

48

2024-10-17T2017T06:2140:2322

2024-10-17T2217T09:4612:3400

In parallel: 3 tenants

All tenants reindex FAILED in
2 hours 25 min "MERGE_FAILED"

5reindex FAILED for 1 tenant [tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]

9

2024-10-16T1415T10:3816:0850

2024-10-16T1515T12:5304:3751

10,099,620

Sequential: fs07000001

1 hour 15 48 min

mod-search:

4.0.0-SNAPSHOT.278

For this version, we encountered out-of-memory (OOM) issues with mod-search and attempted to resolve them by increasing the memory parameters:

Mem Hard Limit

: 4592 MB
Mem Soft Limit

: 4480

6

MB

However, these adjustments did not resolve the issue, so we updated mod-search to version 4.0.0-SNAPSHOT.281.

10

2024-10-16T1616T08:1101:5314

2024-10-16T2116T12:0832:0019

2710,957099,839620

Sequential: fs09000000

4 hours 57 min

7fs07000001

4 hours 31 min

Data nodes instance scaled down to r6g.large.search

This test was performed to check if it is possible to reindex with regular open search size without scaling it.
It is possible to reindex but it will take longer to reindex (approx 4 times longer for r6g.large.search compared to r6g.4xlarge.search)

11

2024-10-17T0621T09:2049:2210

2024-10-17T0621T12:3414:0438

110,210099,000620

Sequential: fs07000002fs07000001

14 min

8

2024-10-17T06:40:22

2024-10-17T09:12:00

In parallel: 3 tenants

reindex FAILED for 1 tenant [tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]

92 hours 25 min

2 instances of database db.r6g.xlarge (reader and writer)

The baseline database size, using the db.r6g.8xlarge instance, is quite large. A performance test was conducted to assess whether this size is optimal. The results indicate that it would be more efficient to use smaller instances, such as db.r6g.4xlarge or db.r6g.2xlarge, instead.

12

2024-10-15T1021T16:1636:5018

2024-10-15T1221T18:0426:5115

10,099,620

Sequential: fs07000001

1 hour 48 50 min

mod-search:4.0.0-SNAPSHOT.278

10

1 instance of database db.r6g.2xlarge

13

2024-10-16T0822T09:0116:1423

2024-10-16T1222T10:3238:1920

10,099,620

Sequential: fs07000001

4 hours 31 min

Data nodes instance scaled down to r6g.large.search

1 hour 22 minutes

1 instance of database db.r6g.4xlarge

Indexing size

All the data from the tables below were captured after each test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

Test #1

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.778Z","endTimeMerge":"2024-10-17T13:08:31.434Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.839Z","endTimeMerge":"2024-10-17T13:13:52.870Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.544Z","endTimeUpload":"2024-10-17T13:50:47.667Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.941Z","endTimeUpload":"2024-10-17T14:02:34.736Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-17T13:14:08.598Z","endTimeUpload":"2024-10-17T14:06:09.460Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:09.193Z","endTimeUpload":"2024-10-17T14:06:46.360Z"}]

Test #2

Code Block

language	json

[{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-17T16:57:51.729Z","endTimeUpload":"2024-10-17T19:49:26.148Z"},
{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:35:59.909Z","endTimeMerge":"2024-10-17T16:42:19.528Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:36:00.032Z","endTimeMerge":"2024-10-17T16:57:34.560Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.682Z","endTimeUpload":"2024-10-17T17:46:13.105Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.850Z","endTimeUpload":"2024-10-17T18:04:25.328Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:52.021Z","endTimeUpload":"2024-10-17T19:03:31.441Z"}]

Test #3

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.803Z","endTimeMerge":"2024-10-17T20:03:45.240Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.813Z","endTimeMerge":"2024-10-17T20:04:19.431Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T20:04:20.952Z","endTimeUpload":"2024-10-17T20:12:07.684Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.959Z","endTimeUpload":"2024-10-17T20:12:13.154Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.986Z","endTimeUpload":"2024-10-17T20:12:17.030Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:21.037Z","endTimeUpload":"2024-10-17T20:12:24.156Z"}]

Test #4

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":57956,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.508Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":55778,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.638Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":55660,"processedMergeRanges":19214,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.324Z","endTimeMerge":"2024-10-17T22:46:34.030Z"}]
[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2970,"processedMergeRanges":2065,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.717Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":20202,"processedMergeRanges":17676,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.700Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":21045,"processedMergeRanges":13092,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.762Z","endTimeMerge":"2024-10-17T20:53:24.774Z"}]
[{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":2423,"processedMergeRanges":2422,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.851Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.877Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.900Z","endTimeMerge":"2024-10-17T20:23:48.044Z"}]

Test #5

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.913Z","endTimeMerge":"2024-10-16T15:04:31.431Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.957Z","endTimeMerge":"2024-10-16T15:07:12.367Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.170Z","endTimeUpload":"2024-10-16T15:40:17.836Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.576Z","endTimeUpload":"2024-10-16T15:51:05.755Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.795Z","endTimeUpload":"2024-10-16T15:53:35.729Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T15:07:26.195Z","endTimeUpload":"2024-10-16T15:53:37.124Z"}]

Test #6

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.185Z","endTimeMerge":"2024-10-16T18:15:03.509Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.305Z","endTimeMerge":"2024-10-16T18:30:07.978Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.240Z","endTimeUpload":"2024-10-16T19:15:39.436Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.398Z","endTimeUpload":"2024-10-16T19:30:23.501Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.522Z","endTimeUpload":"2024-10-16T20:32:58.360Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-16T18:30:22.281Z","endTimeUpload":"2024-10-16T21:08:00.747Z"}]

Test #7

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.898Z","endTimeMerge":"2024-10-17T06:25:56.268Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.905Z","endTimeMerge":"2024-10-17T06:26:35.476Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T06:26:37.038Z","endTimeUpload":"2024-10-17T06:33:52.000Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.241Z","endTimeUpload":"2024-10-17T06:33:56.853Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.481Z","endTimeUpload":"2024-10-17T06:33:57.691Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.807Z","endTimeUpload":"2024-10-17T06:34:04.078Z"}]

Test #9

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.469Z","endTimeMerge":"2024-10-15T10:50:57.419Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.534Z","endTimeMerge":"2024-10-15T10:53:48.693Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.365Z","endTimeUpload":"2024-10-15T11:50:19.948Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.797Z","endTimeUpload":"2024-10-15T12:00:12.641Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-15T10:54:04.383Z","endTimeUpload":"2024-10-15T12:04:14.378Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:05.024Z","endTimeUpload":"2024-10-15T12:04:51.641Z"}]

Test #10

Code Block

language	json

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.876Z","endTimeMerge":"2024-10-16T08:28:09.875Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.935Z","endTimeMerge":"2024-10-16T08:33:38.238Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:42.418Z","endTimeUpload":"2024-10-16T11:52:29.684Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:43.531Z","endTimeUpload":"2024-10-16T12:30:17.561Z:17.561Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T08:33:42.450Z","endTimeUpload":"2024-10-16T12:31:53.638Z"},
{"entityType":"INSTANCECLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":101014095,"processedUploadRanges":101014095,"startTimeUpload":"2024-10-16T08:33:4244.450Z015Z","endTimeUpload":"2024-10-16T12:3132:5319.638Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:44.015Z","endTimeUpload":"2024-10-16T12:32:19.468Z"}]

Compared to Poppy

Compared to the test with instances of 10 mln records size.
Difference in configuration:

Configuration	Ramsons	Poppy
Database	1 instance of db.r6g.8xlarge database instances, writer	2 instances of db.r6g.xlarge database instances, one reader, and one writer
mod-search version	mod-search:4.0.0-SNAPSHOT.281	3.0.0-SNAPSHOT.151
mod-search task count	4	8
Data node instances count (r6g.4xlarge.search)	4	6
Dedicated master nodes	No	3 of r6g.large.search instances

Duration comparison:

...

Ramsons

...

Poppy

...

Delta absolut

...

Delta

...

1 hour 25 min

...

9 hours 38 min

...

8 hours 13 min

...

6.8 times

Resource utilization

Service CPU Utilization

468Z"}]

Compared to Poppy

Compared to the test with instances of 10 mln records size.
Difference in configuration:

Configuration	Ramsons	Poppy
Database	1 instance of db.r6g.8xlarge database instances, writer	2 instances of db.r6g.xlarge database instances, one reader, and one writer
mod-search version	4.0.0-SNAPSHOT.281	3.0.0-SNAPSHOT.151
mod-search task count	4	8
Data node instances count (r6g.4xlarge.search)	4	6
Dedicated master nodes	No	3 of r6g.large.search instances

Duration comparison:

Ramsons

Poppy

Delta absolut

Delta

Ramsons database instance type r6g.8xlarge according to requirements

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-1003

1 hour 25 min

9 hours 38 min

8 hours 13 min

6.8 times

Ramsons: 2 instances of database instance type r6g.xlarge (the same as for Poppy testing)

2 hours 25 min

9 hours 38 min

7 hours 13 min

4 times

Reindex duration and database size correlation:

A larger database instance type typically results in faster reindexing times. However, for the db.r6g.8xlarge and db.r6g.4xlarge, the reindexing duration is nearly identical. Therefore, for 10 million instance records reindex, it's more efficient to use the db.r6g.4xlarge or db.r6g.2xlarge database instance.

Database size	Duration
2 instances of database database db.r6g.xlarge	2 hours 25 min
database db.r6g.2xlarge	1 hour 50 min
database db.r6g.4xlarge	1 hour 22 min
database db.r6g.8xlarge	1 hour 25 min

Resource utilization

Service CPU Utilization

Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage, for all other services CPU did not exceed 20%.

...

Instance CPU Utilization

...

Memory Utilization

DB CPU Utilization

Average database CPU utilization was up to 35%

...

DB Connections

Database use the same average amount of connections

...

Open Search metrics

CPU utilization percentage for all data nodes (Average).

...

Memory usage percentage for all data nodes (Average).

...

Test #11 2 instances of db.r6g.xlarge database: writer and reader instances.

Image Added

Appendix

Infrastructure

PTF-environment rcp1

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instances instance of db.r6g.8xlarge database instances, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch ptf-reindex-testcluster
- OpenSearch version 2.13;
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

Data structure

Module Cluster Resources -	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0

Module versions

Expand

title	All modules and versions

Cluster Resources - rcp1
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.2.1-SNAPSHOT.171	2	4920	4472	1024	3960	512	512
mod-ncip	1	mod-ncip:1.14.6-SNAPSHOT.233	2	1024	896	0	768	88	128
mod-finance-storage	1	mod-finance-storage:8.7.0-SNAPSHOT.181	2	1024	896	1024	700	88	128
mod-agreements	1	mod-agreements:7.1.0-SNAPSHOT.237	2	1592	1488	0	0	0	0
mod-ebsconet	1	mod-ebsconet:2.3.0-SNAPSHOT.80	2	1248	1024	128	700	128	256
mod-organizations	1	mod-organizations:2.0.0-SNAPSHOT.93	2	1024	896	0	700	88	128
edge-sip2	1	edge-sip2:3.3.0-SNAPSHOT.264	2	1024	896	0	768	88	128
mod-settings	1	mod-settings:1.0.4-SNAPSHOT.67	2	1024	896	200	768	88	128
mod-serials-management	1	mod-serials-management:1.1.0-SNAPSHOT.46	2	2480	2312	0	1792	384	512
edge-dematic	1	edge-dematic:2.3.0-SNAPSHOT.143	1	1024	896	0	768	88	128
mod-data-import	1	mod-data-import:3.2.0-SNAPSHOT.189	1	2048	1844	256	1292	384	512
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-inn-reach	1	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	1024	2880	512	1024
mod-record-specifications	10	mod-record-specifications:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-tags	1	mod-tags:2.2.1-SNAPSHOT.138	2	1024	896	0	768	88	128
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.5.0-SNAPSHOT.1160	2	1024	896	0	768	88	128
mod-notify	1	mod-notify:3.2.1-SNAPSHOT.268	2	1024	896	0	768	88	128
mod-inventory-update	1	mod-inventory-update:3.4.2-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-configuration	1	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	1	mod-orders-storage:13.8.0-SNAPSHOT.243	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.3.0-SNAPSHOT.105	2	1024	896	0	768	88	128
mod-login-saml	1	mod-login-saml:2.9.0-SNAPSHOT.291	2	1024	896	0	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:5.0.0-SNAPSHOT.133	2	1024	896	0	768	88	128
mod-password-validator	1	mod-password-validator:3.2.2-SNAPSHOT.149	2	1440	1298	0	768	384	512
mod-gobi	1	mod-gobi:2.9.0-SNAPSHOT.259	2	1024	896	0	700	88	128
mod-licenses	1	mod-licenses:6.1.0-SNAPSHOT.163	2	2480	2312	0	1792	384	512
edge-dcb	1	edge-dcb:1.2.0-SNAPSHOT.39	2	1024	896	0	768	88	128
mod-bulk-operations	1	mod-bulk-operations:2.1.0-SNAPSHOT.119	2	3072	2600	1024	1536	384	512
mod-fqm-manager	10	mod-fqm-manager:2.1.0-SNAPSHOT.1197	2	3000	2600	128	2048	384	512
mod-graphql	1	mod-graphql:1.12.1.412	2	1024	896	0	768	88	128
mod-finance	1	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-erm-usage	1	mod-erm-usage:5.0.0-SNAPSHOT.109	2	1024	896	0	768	88	128
mod-batch-print	1	mod-batch-print:1.2.0-SNAPSHOT.15	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	5	mod-lists:2.1.0-SNAPSHOT.1303	2	1024	896	0	768	88	128
mod-copycat	1	mod-copycat:1.6.1-SNAPSHOT.208	2	1024	512	0	768	88	128
mod-entities-links	4	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	400	1440	0	1024
mod-permissions	2	mod-permissions:6.6.0-SNAPSHOT.385	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	1	mod-orders:12.9.0-SNAPSHOT.398	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.10.2-SNAPSHOT.116	2	1024	896	0	768	88	128
mod-marc-migrations	26	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-inn-reach	1	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	1	mod-users-bl:7.9.0-SNAPSHOT.318	2	1440	1152	512	922	88	128
mod-oa	1	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-invoice	1	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-user-import	1	mod-user-import:3.9.0-SNAPSHOT.197	2	1024	896	0	768	88	128
mod-sender	1	mod-sender:1.12.1-SNAPSHOT.100	2	1024	896	0	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.10.0-SNAPSHOT.174	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.3.0-SNAPSHOT.233	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.7.0-SNAPSHOT.203	2	1024	896	0	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.3.0-SNAPSHOT.405	2	2880	2592	1536	1814	384	512
mod-source-record-storage	2	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	2048	3500	384	512
mod-calendar	2	mod-calendar:3.2.0-SNAPSHOT.229	2	1024	896	0	768	88	128
mod-event-config	1	mod-event-config:2.7.2-SNAPSHOT.110	2	1024	896	0	768	88	128
mod-courses	1	mod-courses:1.4.11-SNAPSHOT.155	2	1024	896	0	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.1-SNAPSHOT.26	2	1024	896	0	0	0	0
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.1-SNAPSHOT.128	2	1024	896	0	768	88	128
mod-requests-mediated	10	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-di-converter-storage	1	mod-di-converter-storage:2.3.0-SNAPSHOT.124	2	1024	896	0	768	88	128
mod-pubsub	1	mod-pubsub:2.15.0-SNAPSHOT.224	2	1536	1440	1024	922	384	512
mod-circulation	1	mod-circulation:24.3.0-SNAPSHOT.681	2	2880	2592	1536	1814	384	512
edge-orders	1	edge-orders:3.1.0-SNAPSHOT.156	2	1024	896	0	768	88	128
edge-rtac	1	edge-rtac:2.7.3-SNAPSHOT.234	2	1024	896	0	768	88	128
mod-template-engine	1	mod-template-engine:1.20.1-SNAPSHOT.118	2	1024	896	0	768	88	128
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.11.0-SNAPSHOT.163	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.1-SNAPSHOT.216	2	1024	896	0	768	88	128
edge-fqm	1	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	1	mod-quick-marc:5.2.0-SNAPSHOT.153	1	2288	2176	0	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.2.0-SNAPSHOT.198	2	1024	896	0	768	88	128
mod-invoice-storage	1	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	1024	1024	384	512
mod-reading-room	2	mod-reading-room:1.0.0-SNAPSHOT.12	2	1024	896	0	768	88	128
mod-service-interaction	1	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.3-SNAPSHOT.40	2	1024	896	0	768	88	128
mod-data-export	1	mod-data-export:5.1.0-SNAPSHOT.240	1	2048	1524	1024	0	0	0
mod-patron	1	mod-patron:6.2.0-SNAPSHOT.165	2	1024	896	0	768	88	128
mod-oai-pmh	1	mod-oai-pmh:3.14.0-SNAPSHOT.402	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.4.0-SNAPSHOT.88	2	1024	896	0	768	88	128
mod-notes	1	mod-notes:5.3.0-SNAPSHOT.325	2	1024	896	0	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.2.0-SNAPSHOT.178	2	1024	896	0	768	88	128
mod-login	1	mod-login:7.12.0-SNAPSHOT.324	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.8.0-SNAPSHOT.131	2	1024	896	0	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.3.1-SNAPSHOT.167	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	1	mod-eusage-reports:3.0.0-SNAPSHOT.73	2	1024	896	0	768	88	128

Methodology/Approach

Use a non-consortium cluster for testing (rcp1 in our case).
Configure the environment in accordance with Infrastructure parameters and requirements that are in the ticket
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key PERF-1003
Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status
- Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/fullwithout any parameters

Script on the http://github.com/folio-org/perf-testing/mod-search

Versions Compared

Old Version 8

New Version Current

Key

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Test Summary

Recommendations & Jiras

Test Runs /Results

Indexing size

Test #1

Test #2

Test #3

Test #4

Test #5

Test #6

Test #7

Test #9

Test #10

Compared to Poppy

Resource utilization

Service CPU Utilization

Compared to Poppy

Resource utilization

Service CPU Utilization

Instance CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Appendix

Infrastructure

Data structure

Module versions

Methodology/Approach

Page Comparison

Versions Compared

Old Version 8

New Version Current

Key

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Test Summary

Recommendations & Jiras

Test Runs /Results

Indexing size

Test #1

Test #2

Test #3

Test #4

Test #5

Test #6

Test #7

Test #9

Test #10

Compared to Poppy

Resource utilization

Service CPU Utilization

Compared to Poppy

Resource utilization

Service CPU Utilization

Instance CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Appendix

Infrastructure

Data structure

Module versions

Methodology/Approach