Table of Contents

...

The document's purpose is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing.

Implementation of the feature

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	UXPROD-4892

Jira ticket:

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-984

...

Reindex could be done in 3 hours and 4 minutes (db.r6g.8xlarge) for 13 million instances for all tenants. It is a new feature and this reindex was started for the central tenant but done for all tenants in parallel. Reindex time matches requirements (Expected response time: Whole reindexing procedure should take under 6 hours ).
Service CPU utilization was up to 60% for mod-search and 5% for mod-inventory-storage. For all other services CPU did not exceed 4%.
Memory utilization was stable and no memory leaks or OOM issues were observed.
RDS CPU utilization was about up to 28% for db.r6g.8xlarge.

Test Runs /Results

Test #

Start time

End time

Instances number

Test Conditions

reindexing on Ramsons release, consortium environment

Duration

Notes

1

2024-10-22T13:02:35

2024-10-22T16:06:18

13,777,503 *

In parallel: all tenants

3 hours 4 minutes

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480
Xmx = -XX:MaxRAMPercentage=85.0

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

...

Subrange of reindexing process from 13:02 - 16:06UTC. A Graph was added to see detailed behavior aggregated on the graph above.

...

CPU utilization percentage for all data nodes

...

Memory usage percentage for all data nodes

Average JVM Memory Pressure

...

Maximum memory utilization (SysMemoryUtilization)

...

Appendix

Infrastructure

PTF-environment rcon

9 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch ptf-reindex-testcluster
- OpenSearch version 2.13
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

Module versions

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

qcon
rcon-pvt
mod-search
2
8
mod-search:
3
4.
2
0.0-SNAPSHOT.281
8
4
2592
2480
2048
1440
-XX:MaxRAMPercentage=85.0
512
1024
FALSE
mod-authtoken
1
3
mod-authtoken:2.
15.1
16.0-SNAPSHOT.303
2
1440
1152
512
0
922
88
128
FALSE
mod-inventory-storage
1
3
mod-inventory-storage:27.
1
2.0-SNAPSHOT.773
2
4
4096
3690
2048
0
3076
384
512
FALSE
1024
mod-inventory
1
2
mod-inventory:20.
2
3.0-SNAPSHOT.546
2
2880
2592
1024
0
1814
384
512
FALSE
mod-users
1
2
mod-users:19.3.
1
3-SNAPSHOT.702
2
1024
896
128
0
768
88
128
FALSE
nginx-okapi
1
2
nginx-okapi:2023.06.14
2
1024
896
128
0
0
0
0
FALSE
okapi-b
1
2
okapi:5.3.0
3
1684
1440
1024
0
922
384
512
FALSE

Expand

title	All modules and versions

Cluster Resources - rcon-pvt (Wed Oct 23 12:15:43 UTC 2024)
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	2	mod-remote-storage:3.2.1-SNAPSHOT.171	2	4920	4472	0	3960	512	512
mod-finance-storage	2	mod-finance-storage:8.7.0-SNAPSHOT.183	2	1024	896	0	700	88	128
mod-ncip	2	mod-ncip:1.14.6-SNAPSHOT.233	2	1024	896	0	768	88	128
mod-agreements	2	mod-agreements:7.1.0-SNAPSHOT.237	2	1592	1488	0	0	0	0
mod-ebsconet	2	mod-ebsconet:2.3.0-SNAPSHOT.80	2	1248	1024	0	700	128	256
mod-organizations	2	mod-organizations:2.0.0-SNAPSHOT.95	2	1024	896	0	700	88	128
mod-consortia	2	mod-consortia:1.2.0-SNAPSHOT.22	2	5136	4776	0	4416	384	512
edge-sip2	2	edge-sip2:3.3.0-SNAPSHOT.264	2	1024	896	0	768	88	128
mod-serials-management	2	mod-serials-management:1.1.0-SNAPSHOT.46	2	2480	2312	0	1792	384	512
mod-settings	2	mod-settings:1.0.4-SNAPSHOT.67	2	1024	896	0	768	88	128
mod-data-import	2	mod-data-import:3.2.0-SNAPSHOT.189	1	2048	1844	0	1292	384	512
mod-search	8	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	0	512	1024
edge-dematic	2	edge-dematic:2.3.0-SNAPSHOT.143	1	1024	896	0	768	88	128
mod-inn-reach	2	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024
mod-record-specifications	2	mod-record-specifications:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-tags	2	mod-tags:2.2.1-SNAPSHOT.138	2	1024	896	0	768	88	128
mod-authtoken	3	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	0	922	88	128
edge-courses	2	edge-courses:1.5.0-SNAPSHOT.1160	2	1024	896	0	768	88	128
mod-notify	2	mod-notify:3.2.1-SNAPSHOT.268	2	1024	896	0	768	88	128
mod-inventory-update	2	mod-inventory-update:3.4.2-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-configuration	2	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	2	mod-orders-storage:13.8.0-SNAPSHOT.245	2	1024	896	0	700	88	128
edge-caiasoft	2	edge-caiasoft:2.3.0-SNAPSHOT.105	2	1024	896	0	768	88	128
mod-login-saml	2	mod-login-saml:2.9.0-SNAPSHOT.291	2	1024	896	0	768	88	128
mod-erm-usage-harvester	2	mod-erm-usage-harvester:5.0.0-SNAPSHOT.133	2	1024	896	0	768	88	128
mod-gobi	2	mod-gobi:2.9.0-SNAPSHOT.259	2	1024	896	0	700	88	128
mod-licenses	2	mod-licenses:6.1.0-SNAPSHOT.163	2	2480	2312	0	1792	384	512
mod-password-validator	2	mod-password-validator:3.2.2-SNAPSHOT.149	2	1440	1298	0	768	384	512
edge-dcb	3	edge-dcb:1.2.0-SNAPSHOT.39	2	1024	896	0	768	88	128
mod-bulk-operations	2	mod-bulk-operations:2.1.0-SNAPSHOT.119	2	3072	2600	0	1536	384	512
mod-fqm-manager	2	mod-fqm-manager:2.1.0-SNAPSHOT.1189	2	3000	2600	0	768	88	128
mod-graphql	2	mod-graphql:1.12.1.412	2	1024	896	0	768	88	128
mod-finance	2	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-erm-usage	2	mod-erm-usage:5.0.0-SNAPSHOT.109	2	2800	2550	0	1800	384	512
mod-batch-print	2	mod-batch-print:1.2.0-SNAPSHOT.15	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	2	mod-lists:2.1.0-SNAPSHOT.1303	2	3000	2600	0	2048	384	512
mod-copycat	2	mod-copycat:1.6.1-SNAPSHOT.208	2	1024	512	0	768	88	128
mod-entities-links	5	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	0	1440	0	1024
mod-permissions	3	mod-permissions:6.6.0-SNAPSHOT.386	2	1684	1544	0	1024	384	512
pub-edge	2	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	2	mod-orders:12.9.0-SNAPSHOT.401	2	2048	1740	0	1024	384	512
edge-patron	2	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	0	768	88	128
mod-marc-migrations	11	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-ncip	3	edge-ncip:1.10.2-SNAPSHOT.116	2	1024	896	0	768	88	128
edge-inn-reach	3	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	2	mod-users-bl:7.9.0-SNAPSHOT.318	2	1440	1152	0	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-inventory-storage	3	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	0	3076	512	1024
mod-invoice	2	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	0	922	88	128
mod-user-import	2	mod-user-import:3.9.0-SNAPSHOT.197	2	1024	896	0	768	88	128
mod-sender	2	mod-sender:1.12.1-SNAPSHOT.100	2	1024	896	0	768	88	128
edge-oai-pmh	2	edge-oai-pmh:2.10.0-SNAPSHOT.174	2	1512	1360	0	1440	384	512
mod-data-export-worker	3	mod-data-export-worker:3.3.0-SNAPSHOT.233	2	3072	2048	0	2048	384	512
mod-rtac	2	mod-rtac:3.7.0-SNAPSHOT.203	2	1024	896	0	768	88	128
mod-circulation-storage	2	mod-circulation-storage:17.3.0-SNAPSHOT.412	2	2880	2592	0	1814	384	512
mod-source-record-storage	3	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	0	3500	384	512
mod-calendar	2	mod-calendar:3.2.0-SNAPSHOT.229	2	1024	896	0	768	88	128
mod-event-config	2	mod-event-config:2.7.2-SNAPSHOT.110	2	1024	896	0	768	88	128
mod-courses	3	mod-courses:1.4.11-SNAPSHOT.155	2	1024	896	0	768	88	128
mod-circulation-item	2	mod-circulation-item:1.0.1-SNAPSHOT.26	2	1024	896	0	0	0	0
mod-inventory	2	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	0	1814	384	512
mod-email	2	mod-email:1.17.1-SNAPSHOT.128	2	2800	2550	0	1800	384	512
mod-requests-mediated	2	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-circulation	2	mod-circulation:24.3.0-SNAPSHOT.685	2	2880	2592	0	1814	384	512
mod-pubsub	2	mod-pubsub:2.15.0-SNAPSHOT.224	2	1536	1440	0	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.3.0-SNAPSHOT.124	2	1024	896	0	768	88	128
edge-rtac	2	edge-rtac:2.7.3-SNAPSHOT.234	2	1024	896	0	768	88	128
edge-orders	2	edge-orders:3.1.0-SNAPSHOT.156	2	1024	896	0	768	88	128
mod-users	2	mod-users:19.3.3-SNAPSHOT.702	2	1024	896	0	768	88	128
mod-template-engine	2	mod-template-engine:1.20.1-SNAPSHOT.118	2	1024	896	0	768	88	128
mod-patron-blocks	2	mod-patron-blocks:1.11.0-SNAPSHOT.163	2	1024	896	0	768	88	128
mod-audit	2	mod-audit:2.9.1-SNAPSHOT.216	2	1024	896	0	768	88	128
edge-fqm	3	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	2	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	0	3500	384	512
nginx-edge	2	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	2	mod-quick-marc:5.2.0-SNAPSHOT.153	1	2288	2176	0	1664	384	512
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	2	okapi:5.3.0	3	1684	1440	0	922	384	512
mod-feesfines	2	mod-feesfines:19.2.0-SNAPSHOT.198	2	1024	896	0	768	88	128
mod-invoice-storage	2	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	0	1024	384	512
mod-reading-room	2	mod-reading-room:1.0.0-SNAPSHOT.12	2	1024	896	0	768	88	128
mod-dcb	2	mod-dcb:1.1.3-SNAPSHOT.40	2	1024	896	0	768	88	128
mod-service-interaction	2	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	0	1290	384	512
mod-patron	2	mod-patron:6.2.0-SNAPSHOT.165	2	1024	896	0	768	88	128
mod-data-export	5	mod-data-export:5.1.0-SNAPSHOT.240	1	2592	2480	0	1440	88	1024
mod-oai-pmh	2	mod-oai-pmh:3.14.0-SNAPSHOT.402	2	4096	3690	0	3076	384	512
edge-connexion	2	edge-connexion:1.4.0-SNAPSHOT.88	2	1024	896	0	768	88	128
mod-notes	2	mod-notes:5.3.0-SNAPSHOT.325	2	1024	896	0	952	384	512
mod-kb-ebsco-java	3	mod-kb-ebsco-java:4.2.0-SNAPSHOT.178	2	1024	896	0	768	88	128
mod-organizations-storage	2	mod-organizations-storage:4.8.0-SNAPSHOT.133	2	1024	896	0	700	88	128
mod-data-export-spring	2	mod-data-export-spring:3.3.1-SNAPSHOT.167	1	2048	1844	0	1536	384	512
mod-login	2	mod-login:7.12.0-SNAPSHOT.326	2	1440	1298	0	768	384	512
pub-okapi	2	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	2	mod-eusage-reports:3.0.0-SNAPSHOT.73	2	1024	896	0	768	88	128

...

Use consortium cluster for testing (rcon in our case).
Configure the environment according to Infrastructure parameters and requirements that are in the ticket
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key PERF-889
Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/fullwithout any parameters on central tenant. For all other tenants in consortium cluster reindex will be performed automatically.
Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status
Script on the http://github.com/folio-org/perf-testing/mod-search

Versions Compared

Old Version 2

New Version Current

Key

Test Runs /Results

Appendix

Infrastructure

Module versions

Page Comparison

Versions Compared

Old Version 2

New Version Current

Key

Test Runs /Results

Appendix

Infrastructure

Module versions