Table of Contents

...

The document's purpose is to assess reindexing performance on a Ramsons release. Calculate reindex time and size of reindexing.

Implementation of the feature

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	UXPROD-4892

Jira ticket:

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-984

Test Summary

...

Reindex could be done in 3 hours and 4 minutes (db.r6g.8xlarge) for 13 million instances for all tenants. It is a new feature and this reindex was started for the central tenant but done for all tenants in parallel. Reindex time matches requirements (Expected response time: Whole reindexing procedure should take under 6 hours ).
Service CPU utilization was up to 60% for mod-search and 5% for mod-inventory-storage. For all other services CPU did not exceed 4%.
Memory utilization was stable and no memory leaks or OOM issues were observed.
RDS CPU utilization was about up to 28% for db.r6g.8xlarge.

Test Runs /Results

Test #

Start time

End time

Total

Instances number

Test Conditions

reindexing on

Poppy

Ramsons release, consortium environment

Duration

Notes

1

2024-10-22T13:02:35

2024-10-22T16:06:18

13,777,503 *

In parallel: all tenants

3 hours 4 minutes

mod-search:

task count = 4
Mem Hard Limit = 2592
Mem Soft Limit = 2480
Xmx = -XX:MaxRAMPercentage=85.0

mod-inventory-storage task count = 4
open search Data nodes instance scaled up to r6g.4xlarge.search

* -Total number of instances for all of the tenants according to the database

Instances number per tenant

Tenant name	Instances number from UI	Instances number from the database
cs00000int	2,216,166	2,216,185
cs00000int_0001	8,799,538	7,015,237
cs00000int_0002	3,560,509	1,347,316
cs00000int_0003	3,187,778	1,135,806
cs00000int_0004	3,038,850	1,054,330
cs00000int_0005	2,836,270	1,004,629

Indexing size

All the data from the table below were captured after the test. Results from request for reindex monitoring GET /search/index/instance-records/reindex/status:

...

Resource utilization

Service CPU Utilization

...

Memory Utilization

Image Added

Instance CPU Utilization

Image Added

DB CPU Utilization

...

DB Connections

...

Open Search metrics

Subrange of reindexing process from 13:02 - 16:06UTC. A Graph was added to see detailed behavior aggregated on the graph above.

...

CPU utilization percentage for all data nodes

...

Memory usage percentage for all data nodes

Average JVM Memory Pressure

...

Maximum memory utilization (SysMemoryUtilization)

...

Appendix

Infrastructure

PTF-environment rcon

9 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.8xlarge database, writer instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch ptf-reindex-testcluster
- OpenSearch version 2.13
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  Enabled - No

...

Module versions

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

qcon-pvt

mod-search

2

mod-search:3.2.0

8

2592

2480

2048

1440

512

1024

FALSE

mod-authtoken

1

mod-authtoken:2.15.1

2

1440

1152

512

922

88

128

FALSE

mod-inventory-storage

1

mod-inventory-storage:27.1.0

2

4096

3690

2048

3076

384

512

FALSE

mod-inventory

1

mod-inventory:20.2.0

2

2880

2592

1024

1814

384

512

FALSE

mod-users

1

mod-users:19.3.1

2

1024

896

128

768

88

128

FALSE

nginx-okapi

1
rcon-pvt
mod-search
8
mod-search:4.0.0-SNAPSHOT.281
4
2592
2480
2048
-XX:MaxRAMPercentage=85.0
512
1024
mod-authtoken
3
mod-authtoken:2.16.0-SNAPSHOT.303
2
1440
1152
0
922
88
128
mod-inventory-storage
3
mod-inventory-storage:27.2.0-SNAPSHOT.773
4
4096
3690
0
3076
512
1024
mod-inventory
2
mod-inventory:20.3.0-SNAPSHOT.546
2
2880
2592
0
1814
384
512
mod-users
2
mod-users:19.3.3-SNAPSHOT.702
2
1024
896
0
768
88
128
nginx-okapi
2
nginx-okapi:2023.06.14
2
1024
896
0
0
0
0
okapi-b
2
okapi:5.3.0
3
1684
1440
0
922
384
512

Expand

title	All modules and versions

Cluster Resources - rcon-pvt (Wed Oct 23 12:15:43 UTC 2024)
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	2	mod-remote-storage:3.2.1-SNAPSHOT.171	2	4920	4472	0	3960	512	512
mod-finance-storage	2	mod-finance-storage:8.7.0-SNAPSHOT.183	2	1024	896	0	700	88	128
mod-ncip	2	mod-ncip:1.14.6-SNAPSHOT.233	2	1024	896	0	768	88	128
mod-agreements	2	mod-agreements:7.1.0-SNAPSHOT.237	2	1592	1488	0	0	0	0
mod-ebsconet	2	mod-ebsconet:2.3.0-SNAPSHOT.80	2	1248	1024	0	700	128	256
mod-organizations	2	mod-organizations:2.0.0-SNAPSHOT.95	2	1024	896	0	700	88	128
mod-consortia	2	mod-consortia:1.2.0-SNAPSHOT.22	2	5136	4776	0	4416	384	512
edge-sip2	2	edge-sip2:3.3.0-SNAPSHOT.264	2	1024	896	0	768	88	128
mod-serials-management	2	mod-serials-management:1.1.0-SNAPSHOT.46	2	2480	2312	0	1792	384	512
mod-settings	2	mod-settings:1.0.4-SNAPSHOT.67	2	1024	896	0	768	88	128
mod-data-import	2	mod-data-import:3.2.0-SNAPSHOT.189	1	2048	1844	0	1292	384	512
mod-search	8	mod-search:4.0.0-SNAPSHOT.281	4	2592	2480	2048	0	512	1024
edge-dematic	2	edge-dematic:2.3.0-SNAPSHOT.143	1	1024	896	0	768	88	128
mod-inn-reach	2	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024
mod-record-specifications	2	mod-record-specifications:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-tags	2	mod-tags:2.2.1-SNAPSHOT.138	2	1024	896	0	768	88	128
mod-authtoken	3	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	0	922	88	128
edge-courses	2	edge-courses:1.5.0-SNAPSHOT.1160	2	1024	896	0	768	88	128
mod-notify	2	mod-notify:3.2.1-SNAPSHOT.268	2	1024	896	0	768	88	128
mod-inventory-update	2	mod-inventory-update:3.4.2-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-configuration	2	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	2	mod-orders-storage:13.8.0-SNAPSHOT.245	2	1024	896	0	700	88	128
edge-caiasoft	2	edge-caiasoft:2.3.0-SNAPSHOT.105	2	1024	896	0	768	88	128
mod-login-saml	2	mod-login-saml:2.9.0-SNAPSHOT.291	2	1024	896	0	768	88	128
mod-erm-usage-harvester	2	mod-erm-usage-harvester:5.0.0-SNAPSHOT.133	2	1024	896	0	768	88	128
mod-gobi	2	mod-gobi:2.9.0-SNAPSHOT.259	2	1024	896	0	700	88	128
mod-licenses	2	mod-licenses:6.1.0-SNAPSHOT.163	2	2480	2312	0	1792	384	512
mod-password-validator	2	mod-password-validator:3.2.2-SNAPSHOT.149	2	1440	1298	0	768	384	512
edge-dcb	3	edge-dcb:1.2.0-SNAPSHOT.39	2	1024	896	0	768	88	128
mod-bulk-operations	2	mod-bulk-operations:2.1.0-SNAPSHOT.119	2	3072	2600	0	1536	384	512
mod-fqm-manager	2	mod-fqm-manager:2.1.0-SNAPSHOT.1189	2	3000	2600	0	768	88	128
mod-graphql	2	mod-graphql:1.12.1.412	2	1024	896	0	768	88	128
mod-finance	2	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-erm-usage	2	mod-erm-usage:5.0.0-SNAPSHOT.109	2	2800	2550	0	1800	384	512
mod-batch-print	2	mod-batch-print:1.2.0-SNAPSHOT.15	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	2	mod-lists:2.1.0-SNAPSHOT.1303	2	3000	2600	0	2048	384	512
mod-copycat	2	mod-copycat:1.6.1-SNAPSHOT.208	2	1024	512	0	768	88	128
mod-entities-links	5	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	0	1440	0	1024
mod-permissions	3	mod-permissions:6.6.0-SNAPSHOT.386	2	1684	1544	0	1024	384	512
pub-edge	2	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	2	mod-orders:12.9.0-SNAPSHOT.401	2	2048	1740	0	1024	384	512
edge-patron	2	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	0	768	88	128
mod-marc-migrations	11	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-ncip	3	edge-ncip:1.10.2-SNAPSHOT.116	2	1024	896	0	768	88	128
edge-inn-reach	3	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	2	mod-users-bl:7.9.0-SNAPSHOT.318	2	1440	1152	0	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-inventory-storage	3	mod-inventory-storage:27.2.0-SNAPSHOT.773	4	4096	3690	0	3076	512	1024
mod-invoice	2	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	0	922	88	128
mod-user-import	2	mod-user-import:3.9.0-SNAPSHOT.197	2	1024	896	0	768	88	128
mod-sender	2	mod-sender:1.12.1-SNAPSHOT.100	2	1024	896	0	768	88	128
edge-oai-pmh	2	edge-oai-pmh:2.10.0-SNAPSHOT.174	2	1512	1360	0	1440	384	512
mod-data-export-worker	3	mod-data-export-worker:3.3.0-SNAPSHOT.233	2	3072	2048	0	2048	384	512
mod-rtac	2	mod-rtac:3.7.0-SNAPSHOT.203	2	1024	896	0	768	88	128
mod-circulation-storage	2	mod-circulation-storage:17.3.0-SNAPSHOT.412	2	2880	2592	0	1814	384	512
mod-source-record-storage	3	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	0	3500	384	512
mod-calendar	2	mod-calendar:3.2.0-SNAPSHOT.229	2	1024	896	0	768	88	128
mod-event-config	2	mod-event-config:2.7.2-SNAPSHOT.110	2	1024	896	0	768	88	128
mod-courses	3	mod-courses:1.4.11-SNAPSHOT.155	2	1024	896	0	768	88	128
mod-circulation-item	2	mod-circulation-item:1.0.1-SNAPSHOT.26	2	1024	896	0	0	0	0
mod-inventory	2	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	0	1814	384	512
mod-email	2	mod-email:1.17.1-SNAPSHOT.128	2	2800	2550	0	1800	384	512
mod-requests-mediated	2	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-circulation	2	mod-circulation:24.3.0-SNAPSHOT.685	2	2880	2592	0	1814	384	512
mod-pubsub	2	mod-pubsub:2.15.0-SNAPSHOT.224	2	1536	1440	0	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.3.0-SNAPSHOT.124	2	1024	896	0	768	88	128
edge-rtac	2	edge-rtac:2.7.3-SNAPSHOT.234	2	1024	896	0	768	88	128
edge-orders	2	edge-orders:3.1.0-SNAPSHOT.156	2	1024	896	0	768	88	128
mod-users	2	mod-users:19.3.3-SNAPSHOT.702	2	1024	896	0	768	88	128
mod-template-engine	2	mod-template-engine:1.20.1-SNAPSHOT.118	2	1024	896	0	768	88	128
mod-patron-blocks	2	mod-patron-blocks:1.11.0-SNAPSHOT.163	2	1024	896	0	768	88	128
mod-audit	2	mod-audit:2.9.1-SNAPSHOT.216	2	1024	896	0	768	88	128
edge-fqm	3	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	2	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	0	3500	384	512
nginx-edge	2	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	2	mod-quick-marc:5.2.0-SNAPSHOT.153	1	2288	2176	0	1664	384	512
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896

...

0

...

okapi-b

...

2	okapi:5.3.0	3	1684	1440	0	922	384	512
mod-feesfines	2	mod-feesfines:19.2.0-SNAPSHOT.198	2	1024	896	0	768	88	128
mod-invoice-storage	2	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	0	1024	384	512
mod-reading-room	2	mod-reading-room:1.0.0-SNAPSHOT.12	2	1024	896	0	768	88	128
mod-dcb	2	mod-dcb:1.1.3-SNAPSHOT.40	2	1024	896	0	768	88	128
mod-service-interaction	2	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	0	1290	384	512
mod-patron	2	mod-patron:6.2.0-SNAPSHOT.165	2	1024	896	0	768	88	128
mod-data-export	5	mod-data-export:5.1.0-SNAPSHOT.240	1	2592	2480	0	1440	88	1024
mod-oai-pmh	2	mod-oai-pmh:3.14.0-SNAPSHOT.402	2	4096	3690	0	3076	384	512
edge-connexion	2	edge-connexion:1.4.0-SNAPSHOT.88	2	1024	896	0	768	88	128
mod-notes	2	mod-notes:5.3.0-SNAPSHOT.325	2	1024	896	0	952	384	512
mod-kb-ebsco-java	3	mod-kb-ebsco-java:4.2.0-SNAPSHOT.178	2	1024	896	0	768	88	128
mod-organizations-storage	2	mod-organizations-storage:4.8.0-SNAPSHOT.133	2	1024

...

922

...

384

...

512

...

FALSE

All modules and versions

Expand

title

	896	0	700	88	128
mod-data-export-spring	2	mod-data-export-spring:3.3.1-SNAPSHOT.167	1	2048	1844	0	1536	384	512
mod-login	2	mod-login:7.12.0-SNAPSHOT.326	2	1440	1298	0	768	384	512
pub-okapi	2	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
mod-eusage-reports	2	mod-eusage-reports:3.0.0-SNAPSHOT.73	2	1024	896	0	768	88	128

Methodology/Approach

Use consortium cluster for testing (rcon in our case).
Configure the environment according to Infrastructure parameters and requirements that are in the ticket
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key PERF-889
Reindex process was started from the JMeter script using POST request /search/index/instance-records/reindex/fullwithout any parameters on central tenant. For all other tenants in consortium cluster reindex will be performed automatically.
Reindex, get the results for indexing time and size from GET /search/index/instance-records/reindex/status
Script on the http://github.com/folio-org/perf-testing/mod-search

Versions Compared

Old Version 1

New Version Current

Key

Test Summary