Overview

The purpose of the document is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing. PERF-1003 - Getting issue details... STATUS

Test Summary

Reindex could be done in 1 hour 25 minutes (db.r6g.8xlarge) for 10 million instances. It is 6.8 times faster than the Poppy release with db.r6g.xlarge. Duration 2 hours 25 minutes with the db.r6g.xlarge (same database size) and it is 4 times faster compared to Poppy.
It is possible to run reindex with the small-size database (xlarge). duration -- hours and we have 10 mln records
It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will fail.
Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage. For all other services CPU did not exceed 20%.
Memory utilization was stable and no memory leaks or OOM issues were observed.
RDS CPU utilization was about 90% for the database db.r6g.xlarge and up to 35% for db.r6g.8xlarge.

Recommendations & Jiras

It is not possible to run multitenant reindex. If starting 3 reindex in parallel for 3 tenants from 1 to 3 reindex will be failed. MSEARCH-868 - Getting issue details... STATUS

Multiple deadlocks in the database were observed at the start of reindex.

It is possible to run reindex on the small-size database (xlarge, 2 instances). duration 2 hours 25 minutes and we have 10 mln records

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480

Xmx = -XX:MaxRAMPercentage=85.0

                    "name": "JAVA_OPTS",
                    "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-search.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Dserver.port=8082 -XX:MaxRAMPercentage=85.0"

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

Test Runs /Results

Test #	Start time	End time	Instances number	Test Conditions reindexing on Poppy release, consortium environment	Duration *	Notes
1	2024-10-17T12:41:14	2024-10-17T14:06:46	10,099,620	Sequential: fs07000001	1 hour 25 min	mod-search: task count = 4 Mem Hard Limit = `2592` Mem Soft Limit = `2480` Xmx = `-XX:MaxRAMPercentage=85.0` mod-inventory-storage task count = 4 open search Data nodes instance scaled up to r6g.4xlarge.search
2	2024-10-17T14:35:59	2024-10-17T19:49:26	27,957,839	Sequential: fs09000000	5 hours 14 min
3	2024-10-17T19:58:07	2024-10-17T20:12:24	1,210,000	Sequential: fs07000002	14 min
4	2024-10-17T20:21:23	2024-10-17T22:46:34		In parallel: 3 tenants	All tenants reindex FAILED in 2 hours 25 min "MERGE_FAILED"
5	2024-10-16T14:38:08	2024-10-16T15:53:37	10,099,620	Sequential: fs07000001	1 hour 15 min	mod-search: task count = 4 Mem Hard Limit = `4592` Mem Soft Limit = `4480` Xmx = `-XX:MaxRAMPercentage=85.0` mod-inventory-storage task count = 4 open search Data nodes instance scaled up to r6g.4xlarge.search
6	2024-10-16T16:11:53	2024-10-16T21:08:00	27,957,839	Sequential: fs09000000	4 hours 57 min
7	2024-10-17T06:20:22	2024-10-17T06:34:04	1,210,000	Sequential: fs07000002	14 min
8	2024-10-17T06:40:22	2024-10-17T09:12:00		In parallel: 3 tenants	reindex FAILED for 1 tenant `[tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]`
9	2024-10-15T10:16:50	2024-10-15T12:04:51	10,099,620	Sequential: fs07000001	1 hour 48 min	mod-search:4.0.0-SNAPSHOT.278
10	2024-10-16T08:01:14	2024-10-16T12:32:19	10,099,620	Sequential: fs07000001	4 hours 31 min	Data nodes instance scaled down to r6g.large.search
11	2024-10-21T09:49:10	2024-10-21T12:14:38	10,099,620	Sequential: fs07000001	2 hours 25 min	2 instances of database db.r6g.xlarge (reader and writer)
12	2024-10-21T16:36:18	2024-10-21T18:26:15	10,099,620	Sequential: fs07000001	1 hour 50 min	1 instance of database db.r6g.2xlarge

Indexing size

All the data from the tables below were captured after each test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

Test #1

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.778Z","endTimeMerge":"2024-10-17T13:08:31.434Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.839Z","endTimeMerge":"2024-10-17T13:13:52.870Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.544Z","endTimeUpload":"2024-10-17T13:50:47.667Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.941Z","endTimeUpload":"2024-10-17T14:02:34.736Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-17T13:14:08.598Z","endTimeUpload":"2024-10-17T14:06:09.460Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:09.193Z","endTimeUpload":"2024-10-17T14:06:46.360Z"}]

Test #2

[{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-17T16:57:51.729Z","endTimeUpload":"2024-10-17T19:49:26.148Z"},
{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:35:59.909Z","endTimeMerge":"2024-10-17T16:42:19.528Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:36:00.032Z","endTimeMerge":"2024-10-17T16:57:34.560Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.682Z","endTimeUpload":"2024-10-17T17:46:13.105Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.850Z","endTimeUpload":"2024-10-17T18:04:25.328Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:52.021Z","endTimeUpload":"2024-10-17T19:03:31.441Z"}]

Test #3

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.803Z","endTimeMerge":"2024-10-17T20:03:45.240Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.813Z","endTimeMerge":"2024-10-17T20:04:19.431Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T20:04:20.952Z","endTimeUpload":"2024-10-17T20:12:07.684Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.959Z","endTimeUpload":"2024-10-17T20:12:13.154Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.986Z","endTimeUpload":"2024-10-17T20:12:17.030Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:21.037Z","endTimeUpload":"2024-10-17T20:12:24.156Z"}]

Test #4

[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":57956,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.508Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":55778,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.638Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":55660,"processedMergeRanges":19214,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.324Z","endTimeMerge":"2024-10-17T22:46:34.030Z"}]
[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2970,"processedMergeRanges":2065,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.717Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":20202,"processedMergeRanges":17676,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.700Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":21045,"processedMergeRanges":13092,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.762Z","endTimeMerge":"2024-10-17T20:53:24.774Z"}]
[{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":2423,"processedMergeRanges":2422,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.851Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.877Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.900Z","endTimeMerge":"2024-10-17T20:23:48.044Z"}]

Test #5

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.913Z","endTimeMerge":"2024-10-16T15:04:31.431Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.957Z","endTimeMerge":"2024-10-16T15:07:12.367Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.170Z","endTimeUpload":"2024-10-16T15:40:17.836Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.576Z","endTimeUpload":"2024-10-16T15:51:05.755Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.795Z","endTimeUpload":"2024-10-16T15:53:35.729Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T15:07:26.195Z","endTimeUpload":"2024-10-16T15:53:37.124Z"}]

Test #6

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.185Z","endTimeMerge":"2024-10-16T18:15:03.509Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.305Z","endTimeMerge":"2024-10-16T18:30:07.978Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.240Z","endTimeUpload":"2024-10-16T19:15:39.436Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.398Z","endTimeUpload":"2024-10-16T19:30:23.501Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.522Z","endTimeUpload":"2024-10-16T20:32:58.360Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-16T18:30:22.281Z","endTimeUpload":"2024-10-16T21:08:00.747Z"}]

Test #7

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.898Z","endTimeMerge":"2024-10-17T06:25:56.268Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.905Z","endTimeMerge":"2024-10-17T06:26:35.476Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T06:26:37.038Z","endTimeUpload":"2024-10-17T06:33:52.000Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.241Z","endTimeUpload":"2024-10-17T06:33:56.853Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.481Z","endTimeUpload":"2024-10-17T06:33:57.691Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.807Z","endTimeUpload":"2024-10-17T06:34:04.078Z"}]

Test #9

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.469Z","endTimeMerge":"2024-10-15T10:50:57.419Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.534Z","endTimeMerge":"2024-10-15T10:53:48.693Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.365Z","endTimeUpload":"2024-10-15T11:50:19.948Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.797Z","endTimeUpload":"2024-10-15T12:00:12.641Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-15T10:54:04.383Z","endTimeUpload":"2024-10-15T12:04:14.378Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:05.024Z","endTimeUpload":"2024-10-15T12:04:51.641Z"}]

Test #10

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.876Z","endTimeMerge":"2024-10-16T08:28:09.875Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.935Z","endTimeMerge":"2024-10-16T08:33:38.238Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:42.418Z","endTimeUpload":"2024-10-16T11:52:29.684Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:43.531Z","endTimeUpload":"2024-10-16T12:30:17.561Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T08:33:42.450Z","endTimeUpload":"2024-10-16T12:31:53.638Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:44.015Z","endTimeUpload":"2024-10-16T12:32:19.468Z"}]

Compared to Poppy

Compared to the test with instances of 10 mln records size.
Difference in configuration:

Configuration	Ramsons	Poppy
Database	1 instance of db.r6g.8xlarge database instances, writer	2 instances of db.r6g.xlarge database instances, one reader, and one writer
mod-search version	mod-search:4.0.0-SNAPSHOT.281	3.0.0-SNAPSHOT.151
mod-search task count	4	8
Data node instances count (r6g.4xlarge.search)	4	6
Dedicated master nodes	No	3 of r6g.large.search instances

Duration comparison:

	Ramsons	Poppy	Delta absolut	Delta
Compared to the database 8xlarge for Ramsons	1 hour 25 min	9 hours 38 min	8 hours 13 min	6.8 times
Compared to 2 instances of database xlarge for Ramsons (the same as for Poppy testing)	2 hours 25 min	9 hours 38 min	7 hours 13 min	4 times

Resource utilization

Service CPU Utilization

Service CPU utilization was up to 50% for mod-search and 40% for mod-inventory-storage. For all other services CPU did not exceed 20%.

Instance CPU Utilization

Memory Utilization

DB CPU Utilization

Average database CPU utilization was up to 35%

DB Connections

Database use the same average amount of connections

Open Search metrics

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Test #11 2 instances of db.r6g.xlarge database: writer and reader instances.

Appendix

Infrastructure

PTF-environment rcp1

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch cluster
- OpenSearch version 2.13;
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

Data structure

Module Cluster Resources -	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0

Module versions

All modules and versions

Cluster Resources - rcp1
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.2.1-SNAPSHOT.171	2	4920	4472	1024	3960	512	512
mod-ncip	1	mod-ncip:1.14.6-SNAPSHOT.233	2	1024	896	0	768	88	128
mod-finance-storage	1	mod-finance-storage:8.7.0-SNAPSHOT.181	2	1024	896	1024	700	88	128
mod-agreements	1	mod-agreements:7.1.0-SNAPSHOT.237	2	1592	1488	0	0	0	0
mod-ebsconet	1	mod-ebsconet:2.3.0-SNAPSHOT.80	2	1248	1024	128	700	128	256
mod-organizations	1	mod-organizations:2.0.0-SNAPSHOT.93	2	1024	896	0	700	88	128
edge-sip2	1	edge-sip2:3.3.0-SNAPSHOT.264	2	1024	896	0	768	88	128
mod-settings	1	mod-settings:1.0.4-SNAPSHOT.67	2	1024	896	200	768	88	128
mod-serials-management	1	mod-serials-management:1.1.0-SNAPSHOT.46	2	2480	2312	0	1792	384	512
edge-dematic	1	edge-dematic:2.3.0-SNAPSHOT.143	1	1024	896	0	768	88	128
mod-data-import	1	mod-data-import:3.2.0-SNAPSHOT.189	1	2048	1844	256	1292	384	512
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-inn-reach	1	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	1024	2880	512	1024
mod-record-specifications	10	mod-record-specifications:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-tags	1	mod-tags:2.2.1-SNAPSHOT.138	2	1024	896	0	768	88	128
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.5.0-SNAPSHOT.1160	2	1024	896	0	768	88	128
mod-notify	1	mod-notify:3.2.1-SNAPSHOT.268	2	1024	896	0	768	88	128
mod-inventory-update	1	mod-inventory-update:3.4.2-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-configuration	1	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	1	mod-orders-storage:13.8.0-SNAPSHOT.243	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.3.0-SNAPSHOT.105	2	1024	896	0	768	88	128
mod-login-saml	1	mod-login-saml:2.9.0-SNAPSHOT.291	2	1024	896	0	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:5.0.0-SNAPSHOT.133	2	1024	896	0	768	88	128
mod-password-validator	1	mod-password-validator:3.2.2-SNAPSHOT.149	2	1440	1298	0	768	384	512
mod-gobi	1	mod-gobi:2.9.0-SNAPSHOT.259	2	1024	896	0	700	88	128
mod-licenses	1	mod-licenses:6.1.0-SNAPSHOT.163	2	2480	2312	0	1792	384	512
edge-dcb	1	edge-dcb:1.2.0-SNAPSHOT.39	2	1024	896	0	768	88	128
mod-bulk-operations	1	mod-bulk-operations:2.1.0-SNAPSHOT.119	2	3072	2600	1024	1536	384	512
mod-fqm-manager	10	mod-fqm-manager:2.1.0-SNAPSHOT.1197	2	3000	2600	128	2048	384	512
mod-graphql	1	mod-graphql:1.12.1.412	2	1024	896	0	768	88	128
mod-finance	1	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-erm-usage	1	mod-erm-usage:5.0.0-SNAPSHOT.109	2	1024	896	0	768	88	128
mod-batch-print	1	mod-batch-print:1.2.0-SNAPSHOT.15	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	5	mod-lists:2.1.0-SNAPSHOT.1303	2	1024	896	0	768	88	128
mod-copycat	1	mod-copycat:1.6.1-SNAPSHOT.208	2	1024	512	0	768	88	128
mod-entities-links	4	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	400	1440	0	1024
mod-permissions	2	mod-permissions:6.6.0-SNAPSHOT.385	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	1	mod-orders:12.9.0-SNAPSHOT.398	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.10.2-SNAPSHOT.116	2	1024	896	0	768	88	128
mod-marc-migrations	26	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-inn-reach	1	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	1	mod-users-bl:7.9.0-SNAPSHOT.318	2	1440	1152	512	922	88	128
mod-oa	1	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-invoice	1	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-user-import	1	mod-user-import:3.9.0-SNAPSHOT.197	2	1024	896	0	768	88	128
mod-sender	1	mod-sender:1.12.1-SNAPSHOT.100	2	1024	896	0	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.10.0-SNAPSHOT.174	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.3.0-SNAPSHOT.233	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.7.0-SNAPSHOT.203	2	1024	896	0	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.3.0-SNAPSHOT.405	2	2880	2592	1536	1814	384	512
mod-source-record-storage	2	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	2048	3500	384	512
mod-calendar	2	mod-calendar:3.2.0-SNAPSHOT.229	2	1024	896	0	768	88	128
mod-event-config	1	mod-event-config:2.7.2-SNAPSHOT.110	2	1024	896	0	768	88	128
mod-courses	1	mod-courses:1.4.11-SNAPSHOT.155	2	1024	896	0	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.1-SNAPSHOT.26	2	1024	896	0	0	0	0
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.1-SNAPSHOT.128	2	1024	896	0	768	88	128
mod-requests-mediated	10	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-di-converter-storage	1	mod-di-converter-storage:2.3.0-SNAPSHOT.124	2	1024	896	0	768	88	128
mod-pubsub	1	mod-pubsub:2.15.0-SNAPSHOT.224	2	1536	1440	1024	922	384	512
mod-circulation	1	mod-circulation:24.3.0-SNAPSHOT.681	2	2880	2592	1536	1814	384	512
edge-orders	1	edge-orders:3.1.0-SNAPSHOT.156	2	1024	896	0	768	88	128
edge-rtac	1	edge-rtac:2.7.3-SNAPSHOT.234	2	1024	896	0	768	88	128
mod-template-engine	1	mod-template-engine:1.20.1-SNAPSHOT.118	2	1024	896	0	768	88	128
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.11.0-SNAPSHOT.163	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.1-SNAPSHOT.216	2	1024	896	0	768	88	128
edge-fqm	1	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	1	mod-quick-marc:5.2.0-SNAPSHOT.153	1	2288	2176	0	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.2.0-SNAPSHOT.198	2	1024	896	0	768	88	128
mod-invoice-storage	1	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	1024	1024	384	512
mod-reading-room	2	mod-reading-room:1.0.0-SNAPSHOT.12	2	1024	896	0	768	88	128
mod-service-interaction	1	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.3-SNAPSHOT.40	2	1024	896	0	768	88	128
mod-data-export	1	mod-data-export:5.1.0-SNAPSHOT.240	1	2048	1524	1024	0	0	0
mod-patron	1	mod-patron:6.2.0-SNAPSHOT.165	2	1024	896	0	768	88	128
mod-oai-pmh	1	mod-oai-pmh:3.14.0-SNAPSHOT.402	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.4.0-SNAPSHOT.88	2	1024	896	0	768	88	128
mod-notes	1	mod-notes:5.3.0-SNAPSHOT.325	2	1024	896	0	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.2.0-SNAPSHOT.178	2	1024	896	0	768	88	128
mod-login	1	mod-login:7.12.0-SNAPSHOT.324	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.8.0-SNAPSHOT.131	2	1024	896	0	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.3.1-SNAPSHOT.167	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	1	mod-eusage-reports:3.0.0-SNAPSHOT.73	2	1024	896	0	768	88	128

Methodology/Approach

Use a non-consortium cluster for testing (rcp1 in our case).
Configure the environment in accordance with Infrastructure parameters and requirements that are in the ticket PERF-1003 - Getting issue details... STATUS
Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status
- Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/full without any parameters

mod-search: Test Reindexing full (Ramsons)