Overview

The purpose of the document is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing. PERF-1003 - Getting issue details... STATUS

Recommendations & Jiras

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480

Xmx = -XX:MaxRAMPercentage=85.0

                    "name": "JAVA_OPTS",
                    "value": "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-search.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Dserver.port=8082 -XX:MaxRAMPercentage=85.0"

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search
1 instance of db.r6g.8xlarge database instances, writer

Test Summary

Test Runs /Results

Test #	Start time	End time	Instances number	Test Conditions reindexing on Poppy release, consortium environment	Duration *	Notes
1	2024-10-17T12:41:14	2024-10-17T14:06:46	10,099,620	Sequential: fs07000001	1 hour 25 min	mod-search: task count = 4 Mem Hard Limit = `2592` Mem Soft Limit = `2480` Xmx = `-XX:MaxRAMPercentage=85.0` mod-inventory-storage task count = 4 open search Data nodes instance scaled up to r6g.4xlarge.search
2	2024-10-17T14:35:59	2024-10-17T19:49:26	27,957,839	Sequential: fs09000000	5 hours 14 min
3	2024-10-17T19:58:07	2024-10-17T20:12:24	1,210,000	Sequential: fs07000002	14 min
4	2024-10-17T20:21:23	2024-10-17T22:46:34		In parallel: 3 tenants	All tenants reindex FAILED in 2 hours 25 min "MERGE_FAILED"
5	2024-10-16T14:38:08	2024-10-16T15:53:37	10,099,620	Sequential: fs07000001	1 hour 15 min	mod-search: task count = 4 Mem Hard Limit = `4592` Mem Soft Limit = `4480` Xmx = `-XX:MaxRAMPercentage=85.0` mod-inventory-storage task count = 4 open search Data nodes instance scaled up to r6g.4xlarge.search
6	2024-10-16T16:11:53	2024-10-16T21:08:00	27,957,839	Sequential: fs09000000	4 hours 57 min
7	2024-10-17T06:20:22	2024-10-17T06:34:04	1,210,000	Sequential: fs07000002	14 min
8	2024-10-17T06:40:22	2024-10-17T09:12:00		In parallel: 3 tenants	reindex FAILED for 1 tenant `[tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]`
9	2024-10-15T10:16:50	2024-10-15T12:04:51	10,099,620	Sequential: fs07000001	1 hour 48 min	mod-search:4.0.0-SNAPSHOT.278
10	2024-10-16T08:01:14	2024-10-16T12:32:19	10,099,620	Sequential: fs07000001	4 hours 31 min	Data nodes instance scaled down to r6g.large.search

Indexing size

All the data from the tables below were captured after each test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

Test #1

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.778Z","endTimeMerge":"2024-10-17T13:08:31.434Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T12:41:14.839Z","endTimeMerge":"2024-10-17T13:13:52.870Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.544Z","endTimeUpload":"2024-10-17T13:50:47.667Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:08.941Z","endTimeUpload":"2024-10-17T14:02:34.736Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-17T13:14:08.598Z","endTimeUpload":"2024-10-17T14:06:09.460Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T13:14:09.193Z","endTimeUpload":"2024-10-17T14:06:46.360Z"}]

Test #2

[{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-17T16:57:51.729Z","endTimeUpload":"2024-10-17T19:49:26.148Z"},
{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:35:59.909Z","endTimeMerge":"2024-10-17T16:42:19.528Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T14:36:00.032Z","endTimeMerge":"2024-10-17T16:57:34.560Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.682Z","endTimeUpload":"2024-10-17T17:46:13.105Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:51.850Z","endTimeUpload":"2024-10-17T18:04:25.328Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T16:57:52.021Z","endTimeUpload":"2024-10-17T19:03:31.441Z"}]

Test #3

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.803Z","endTimeMerge":"2024-10-17T20:03:45.240Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T19:58:07.813Z","endTimeMerge":"2024-10-17T20:04:19.431Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T20:04:20.952Z","endTimeUpload":"2024-10-17T20:12:07.684Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.959Z","endTimeUpload":"2024-10-17T20:12:13.154Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:20.986Z","endTimeUpload":"2024-10-17T20:12:17.030Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T20:04:21.037Z","endTimeUpload":"2024-10-17T20:12:24.156Z"}]

Test #4

[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":57956,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.508Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":55778,"processedMergeRanges":0,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.638Z","endTimeMerge":"2024-10-17T22:46:34.030Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":55660,"processedMergeRanges":19214,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:22:02.324Z","endTimeMerge":"2024-10-17T22:46:34.030Z"}]
[{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2970,"processedMergeRanges":2065,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.717Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":20202,"processedMergeRanges":17676,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.700Z","endTimeMerge":"2024-10-17T20:53:24.774Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":21045,"processedMergeRanges":13092,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:24.762Z","endTimeMerge":"2024-10-17T20:53:24.774Z"}]
[{"entityType":"INSTANCE","status":"MERGE_FAILED","totalMergeRanges":2423,"processedMergeRanges":2422,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.851Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"ITEM","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.877Z","endTimeMerge":"2024-10-17T20:23:48.044Z"},{"entityType":"HOLDINGS","status":"MERGE_FAILED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T20:21:23.900Z","endTimeMerge":"2024-10-17T20:23:48.044Z"}]

Test #5

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.913Z","endTimeMerge":"2024-10-16T15:04:31.431Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T14:38:08.957Z","endTimeMerge":"2024-10-16T15:07:12.367Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.170Z","endTimeUpload":"2024-10-16T15:40:17.836Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.576Z","endTimeUpload":"2024-10-16T15:51:05.755Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T15:07:26.795Z","endTimeUpload":"2024-10-16T15:53:35.729Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T15:07:26.195Z","endTimeUpload":"2024-10-16T15:53:37.124Z"}]

Test #6

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":57956,"processedMergeRanges":57956,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.185Z","endTimeMerge":"2024-10-16T18:15:03.509Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":55778,"processedMergeRanges":55778,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T16:11:53.305Z","endTimeMerge":"2024-10-16T18:30:07.978Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.240Z","endTimeUpload":"2024-10-16T19:15:39.436Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.398Z","endTimeUpload":"2024-10-16T19:30:23.501Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T18:30:22.522Z","endTimeUpload":"2024-10-16T20:32:58.360Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":27830,"processedUploadRanges":27830,"startTimeUpload":"2024-10-16T18:30:22.281Z","endTimeUpload":"2024-10-16T21:08:00.747Z"}]

Test #7

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.898Z","endTimeMerge":"2024-10-17T06:25:56.268Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":2408,"processedMergeRanges":2408,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-17T06:20:22.905Z","endTimeMerge":"2024-10-17T06:26:35.476Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":1212,"processedUploadRanges":1212,"startTimeUpload":"2024-10-17T06:26:37.038Z","endTimeUpload":"2024-10-17T06:33:52.000Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.241Z","endTimeUpload":"2024-10-17T06:33:56.853Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.481Z","endTimeUpload":"2024-10-17T06:33:57.691Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-17T06:26:37.807Z","endTimeUpload":"2024-10-17T06:34:04.078Z"}]

Test #9

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.469Z","endTimeMerge":"2024-10-15T10:50:57.419Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-15T10:16:50.534Z","endTimeMerge":"2024-10-15T10:53:48.693Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.365Z","endTimeUpload":"2024-10-15T11:50:19.948Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:04.797Z","endTimeUpload":"2024-10-15T12:00:12.641Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-15T10:54:04.383Z","endTimeUpload":"2024-10-15T12:04:14.378Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-15T10:54:05.024Z","endTimeUpload":"2024-10-15T12:04:51.641Z"}]

Test #10

[{"entityType":"ITEM","status":"MERGE_COMPLETED","totalMergeRanges":2970,"processedMergeRanges":2970,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.876Z","endTimeMerge":"2024-10-16T08:28:09.875Z"},
{"entityType":"HOLDINGS","status":"MERGE_COMPLETED","totalMergeRanges":21045,"processedMergeRanges":21045,"totalUploadRanges":0,"processedUploadRanges":0,"startTimeMerge":"2024-10-16T08:01:14.935Z","endTimeMerge":"2024-10-16T08:33:38.238Z"},
{"entityType":"SUBJECT","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:42.418Z","endTimeUpload":"2024-10-16T11:52:29.684Z"},
{"entityType":"CONTRIBUTOR","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:43.531Z","endTimeUpload":"2024-10-16T12:30:17.561Z"},
{"entityType":"INSTANCE","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":10101,"processedUploadRanges":10101,"startTimeUpload":"2024-10-16T08:33:42.450Z","endTimeUpload":"2024-10-16T12:31:53.638Z"},
{"entityType":"CLASSIFICATION","status":"UPLOAD_COMPLETED","totalMergeRanges":0,"processedMergeRanges":0,"totalUploadRanges":4095,"processedUploadRanges":4095,"startTimeUpload":"2024-10-16T08:33:44.015Z","endTimeUpload":"2024-10-16T12:32:19.468Z"}]

Compared to Poppy

Compared to the test with instances of 10 mln records size.
Difference in configuration:

Configuration	Ramsons	Poppy
Database	1 instance of db.r6g.8xlarge database instances, writer	2 instances of db.r6g.xlarge database instances, one reader, and one writer
mod-search version	mod-search:4.0.0-SNAPSHOT.281	3.0.0-SNAPSHOT.151
mod-search task count	4	8
Data node instances count (r6g.4xlarge.search)	4	6
Dedicated master nodes	No	3 of r6g.large.search instances

Duration comparison:

Ramsons	Poppy	Delta absolut	Delta
1 hour 25 min	9 hours 38 min	8 hours 13 min	6.8 times

Resource utilization

Service CPU Utilization

Instance CPU Utilization

Memory Utilization

DB CPU Utilization

Average database CPU utilization was up to 35%

DB Connections

Database use the same average amount of connections

Open Search metrics

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Appendix

Infrastructure

PTF-environment rcp1

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instances of db.r6g.8xlarge database instances.
MSK
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch cluster
- OpenSearch version 2.13;
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

Data structure

Module Cluster Resources -	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0

Module versions

All modules and versions

Cluster Resources - rcp1
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.2.1-SNAPSHOT.171	2	4920	4472	1024	3960	512	512
mod-ncip	1	mod-ncip:1.14.6-SNAPSHOT.233	2	1024	896	0	768	88	128
mod-finance-storage	1	mod-finance-storage:8.7.0-SNAPSHOT.181	2	1024	896	1024	700	88	128
mod-agreements	1	mod-agreements:7.1.0-SNAPSHOT.237	2	1592	1488	0	0	0	0
mod-ebsconet	1	mod-ebsconet:2.3.0-SNAPSHOT.80	2	1248	1024	128	700	128	256
mod-organizations	1	mod-organizations:2.0.0-SNAPSHOT.93	2	1024	896	0	700	88	128
edge-sip2	1	edge-sip2:3.3.0-SNAPSHOT.264	2	1024	896	0	768	88	128
mod-settings	1	mod-settings:1.0.4-SNAPSHOT.67	2	1024	896	200	768	88	128
mod-serials-management	1	mod-serials-management:1.1.0-SNAPSHOT.46	2	2480	2312	0	1792	384	512
edge-dematic	1	edge-dematic:2.3.0-SNAPSHOT.143	1	1024	896	0	768	88	128
mod-data-import	1	mod-data-import:3.2.0-SNAPSHOT.189	1	2048	1844	256	1292	384	512
mod-search	10	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	MaxRAMPercentage=85.0	512	1024
mod-inn-reach	1	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	1024	2880	512	1024
mod-record-specifications	10	mod-record-specifications:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-tags	1	mod-tags:2.2.1-SNAPSHOT.138	2	1024	896	0	768	88	128
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.5.0-SNAPSHOT.1160	2	1024	896	0	768	88	128
mod-notify	1	mod-notify:3.2.1-SNAPSHOT.268	2	1024	896	0	768	88	128
mod-inventory-update	1	mod-inventory-update:3.4.2-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-configuration	1	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	1	mod-orders-storage:13.8.0-SNAPSHOT.243	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.3.0-SNAPSHOT.105	2	1024	896	0	768	88	128
mod-login-saml	1	mod-login-saml:2.9.0-SNAPSHOT.291	2	1024	896	0	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:5.0.0-SNAPSHOT.133	2	1024	896	0	768	88	128
mod-password-validator	1	mod-password-validator:3.2.2-SNAPSHOT.149	2	1440	1298	0	768	384	512
mod-gobi	1	mod-gobi:2.9.0-SNAPSHOT.259	2	1024	896	0	700	88	128
mod-licenses	1	mod-licenses:6.1.0-SNAPSHOT.163	2	2480	2312	0	1792	384	512
edge-dcb	1	edge-dcb:1.2.0-SNAPSHOT.39	2	1024	896	0	768	88	128
mod-bulk-operations	1	mod-bulk-operations:2.1.0-SNAPSHOT.119	2	3072	2600	1024	1536	384	512
mod-fqm-manager	10	mod-fqm-manager:2.1.0-SNAPSHOT.1197	2	3000	2600	128	2048	384	512
mod-graphql	1	mod-graphql:1.12.1.412	2	1024	896	0	768	88	128
mod-finance	1	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-erm-usage	1	mod-erm-usage:5.0.0-SNAPSHOT.109	2	1024	896	0	768	88	128
mod-batch-print	1	mod-batch-print:1.2.0-SNAPSHOT.15	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	5	mod-lists:2.1.0-SNAPSHOT.1303	2	1024	896	0	768	88	128
mod-copycat	1	mod-copycat:1.6.1-SNAPSHOT.208	2	1024	512	0	768	88	128
mod-entities-links	4	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	400	1440	0	1024
mod-permissions	2	mod-permissions:6.6.0-SNAPSHOT.385	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	1	mod-orders:12.9.0-SNAPSHOT.398	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.10.2-SNAPSHOT.116	2	1024	896	0	768	88	128
mod-marc-migrations	26	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-inn-reach	1	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	1	mod-users-bl:7.9.0-SNAPSHOT.318	2	1440	1152	512	922	88	128
mod-oa	1	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-invoice	1	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	2048	3076	512	1024
mod-user-import	1	mod-user-import:3.9.0-SNAPSHOT.197	2	1024	896	0	768	88	128
mod-sender	1	mod-sender:1.12.1-SNAPSHOT.100	2	1024	896	0	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.10.0-SNAPSHOT.174	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.3.0-SNAPSHOT.233	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.7.0-SNAPSHOT.203	2	1024	896	0	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.3.0-SNAPSHOT.405	2	2880	2592	1536	1814	384	512
mod-source-record-storage	2	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	2048	3500	384	512
mod-calendar	2	mod-calendar:3.2.0-SNAPSHOT.229	2	1024	896	0	768	88	128
mod-event-config	1	mod-event-config:2.7.2-SNAPSHOT.110	2	1024	896	0	768	88	128
mod-courses	1	mod-courses:1.4.11-SNAPSHOT.155	2	1024	896	0	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.1-SNAPSHOT.26	2	1024	896	0	0	0	0
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.1-SNAPSHOT.128	2	1024	896	0	768	88	128
mod-requests-mediated	10	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-di-converter-storage	1	mod-di-converter-storage:2.3.0-SNAPSHOT.124	2	1024	896	0	768	88	128
mod-pubsub	1	mod-pubsub:2.15.0-SNAPSHOT.224	2	1536	1440	1024	922	384	512
mod-circulation	1	mod-circulation:24.3.0-SNAPSHOT.681	2	2880	2592	1536	1814	384	512
edge-orders	1	edge-orders:3.1.0-SNAPSHOT.156	2	1024	896	0	768	88	128
edge-rtac	1	edge-rtac:2.7.3-SNAPSHOT.234	2	1024	896	0	768	88	128
mod-template-engine	1	mod-template-engine:1.20.1-SNAPSHOT.118	2	1024	896	0	768	88	128
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.11.0-SNAPSHOT.163	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.1-SNAPSHOT.216	2	1024	896	0	768	88	128
edge-fqm	1	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	1	mod-quick-marc:5.2.0-SNAPSHOT.153	1	2288	2176	0	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.2.0-SNAPSHOT.198	2	1024	896	0	768	88	128
mod-invoice-storage	1	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	1024	1024	384	512
mod-reading-room	2	mod-reading-room:1.0.0-SNAPSHOT.12	2	1024	896	0	768	88	128
mod-service-interaction	1	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.3-SNAPSHOT.40	2	1024	896	0	768	88	128
mod-data-export	1	mod-data-export:5.1.0-SNAPSHOT.240	1	2048	1524	1024	0	0	0
mod-patron	1	mod-patron:6.2.0-SNAPSHOT.165	2	1024	896	0	768	88	128
mod-oai-pmh	1	mod-oai-pmh:3.14.0-SNAPSHOT.402	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.4.0-SNAPSHOT.88	2	1024	896	0	768	88	128
mod-notes	1	mod-notes:5.3.0-SNAPSHOT.325	2	1024	896	0	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.2.0-SNAPSHOT.178	2	1024	896	0	768	88	128
mod-login	1	mod-login:7.12.0-SNAPSHOT.324	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.8.0-SNAPSHOT.131	2	1024	896	0	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.3.1-SNAPSHOT.167	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	1	mod-eusage-reports:3.0.0-SNAPSHOT.73	2	1024	896	0	768	88	128

Methodology/Approach

Use consortium cluster for testing (qcon in our case).
Configure the environment in accordance with Infrastructure parameters and requirements that are in the ticket PERF-889 - Getting issue details... STATUS
Reindex, get the results for indexing time, and index size. Use Steps for testing process#Reindex for details.
- Reindex process was started from the JMeter script using POST request /search/index/inventory/reindex with parameters

{ "recreateIndex": true,
"resourceName": "instance" }

mod-search: Test Reindexing full (Ramsons)