1 Overview
2 Recommendations & Jiras
3 Test Summary
4 Test Runs /Results
- 4.1 Indexing size
5 Resource utilization
- 5.1 Test #1 (cs00000int main tenant)
  - 5.1.1 Service CPU Utilization
  - 5.1.2 Memory Utilization
  - 5.1.3 DB CPU Utilization
  - 5.1.4 DB Connections
  - 5.1.5 Open Search metrics
- 5.2 Test #2 (cs00000intt_0001)
  - 5.2.1 Memory Utilization
  - 5.2.2 DB CPU Utilization
  - 5.2.3 DB Connections
- 5.3 Test #3 (cs00000intt_0002)
  - 5.3.1 Service CPU Utilization
  - 5.3.2 Memory Utilization
  - 5.3.3 DB CPU Utilization
  - 5.3.4 DB Connections
  - 5.3.5 Open Search metrics
- 5.4 Test #4 (In parallel: 3 tenants)
  - 5.4.1 Service CPU Utilization
  - 5.4.2 Memory Utilization
  - 5.4.3 DB CPU Utilization
  - 5.4.4 DB Connections
  - 5.4.5 Open Search metrics
6 Appendix
- 6.1 Infrastructure
- 6.2 Data structure
- 6.3 Module versions
- 6.4 Methodology/Approach

Overview

The purpose of the document is to assess reindexing performance on a consortium environment with Quesnelia release. Calculate reindex time and size of reindexing.

Recommendations & Jiras

Original ticket to test: PERF-889: ECS Reindex for mod-search (classification browse)Closed
Additional info ElasticSearch Reindex Performance Recommendations

Test Summary

Test Runs /Results

Test #	Instances number	Test Conditions reindexing on Poppy release, consortium environment	Duration *	Notes
2024-04-24 09:37-12.10	1,706,932	Sequential: cs00000int	2 hours 23 min	mod-search task count =8 mod-inventory-storage task count = 2 mod-okapi task count = 3 open search Data nodes instance scaled up to r6g.4xlarge.search without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch
2024-04-24 13:03 - 23:38	6,905,646	Sequential: cs00000int_0001	10 hours 35 min
3. 2024-04-25 00:19 - 12:28	6,937,091	Sequential: cs00000int_0002	12 hours
4. 2024-04-25 11:45 - 2024-04-26 04:30		In parallel: 3 tenants	16 hours 45 minutes

Indexing size

All the data from the tables below were captured after each test was finished. Results from get-request for reindex monitoring https://127.0.0.1:9999/_cat/indices:

In parallel: 3 tenants

In parallel: 3 tenants
health	status	index	uuid	pri	rep	docs. count	docs. deleted	store.size	pri.store. size
green	open	qcon_instance_subject_cs00000int	nwbohqi6SGiQsrlPjMOlOg	4	2	936066	87976	8gb	2.6gb
green	open	qcon_contributor_cs00000int	UemRKKfuTjSV4JrzZLQEGg	4	2	880696	77071	8.7gb	2.9gb
green	open	qcon_instance_classification_cs00000int	ixObbJPQTJyAHelpcV-r8w	4	2	0	0	2.4kb	832b
green	open	qcon_instance_cs00000int	NnavdW7vSc-2MH2Hvy5K-A	4	2	4929451	180596	62.3gb	20.7gb

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
Sequential: cs00000int
green	open	qcon_instance_subject_cs00000int	Sf1cHAdHQ4G1qCHzAWxHJg	4	2	904477	195457	4.5gb	1.6gb
green	open	qcon_contributor_cs00000int	pzG1jGK3TdmG--jQXucPbw	4	2	846936	114235	3.7gb	1.1gb
green	open	qcon_instance_classification_cs00000int	cumgAMEFQFaFHRGQQfG2-w	4	2	0	0	2.4kb	832b
green	open	qcon_instance_cs00000int	yEg2R1xERAGZBTPvFkVeSA	4	2	1706932	0	31.4gb	10.4gb
Sequential: cs00000int_0001
green	open	qcon_instance_subject_cs00000int	Sf1cHAdHQ4G1qCHzAWxHJg	4	2	906513	98723	9.9gb	3.1gb
green	open	qcon_contributor_cs00000int	pzG1jGK3TdmG--jQXucPbw	4	2	849593	22767	7.9gb	2.8gb
green	open	qcon_instance_classification_cs00000int	cumgAMEFQFaFHRGQQfG2	4	2	0	0	2.4kb	832b
green	open	qcon_instance_cs00000int	yEg2R1xERAGZBTPvFkVeSA	4	2	6905646	216733	74.6gb	24.5gb
Sequential: cs00000int_0002
green	open	qcon_instance_subject_cs00000int	Sf1cHAdHQ4G1qCHzAWxHJg	4	2	909754	69518	13.6gb	4.2gb
green	open	qcon_contributor_cs00000int	pzG1jGK3TdmG--jQXucPbw	4	2	852873	30507	8.9gb	3.3gb
green	open	qcon_instance_classification_cs00000int	cumgAMEFQFaFHRGQQfG2-w	4	2	0	0	2.4kb	832b
green	open	qcon_instance_cs00000int	yEg2R1xERAGZBTPvFkVeSA	4	2	6937091	257165	74.7gb	24.8gb

Resource utilization

Test #1 (cs00000int main tenant)

During the test on cs00000int tenant, service CPU Utilization was stable and consistent. Most consuming services were mod-inventory storage = 94%, nginx-okapi = 81%, okapi = 60%, mod-inventory =15%. No memory leaks. Database connection increased by 50 and was equal to 650. DB CPU usage increased from 1 to 14% during the first 15 minutes of reindexing, this spike correlated with the changing of service CPU utilization and was caused by a high indexing data rate(about 500K operation per minute).

Open search metrics.

At the beginning of the reindex data rate was about 500K operations per minute. After 15 reindex data rate decreased to 8K. Indexing latency increased after 15 minutes the test started and was about 4 milliseconds and after 50 minutes decreased to 1.5 milliseconds. This spike correlates with the increasing of data nodes CPU utilization up to 16 % and next decreasing to 7%. CPU Utilization of the master node was changed from 5 to 27% during reindexing.

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Indexing Data Rate

Subrange of reindexing process from 09:55- 11:50. A Graph was added to see detailed behavior aggregated on the graph above.

Indexing latency

Master CPU Utilization (Average)

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Test #2 (cs00000intt_0001)

During the test on cs00000intt_0001 tenant, the Most consuming CPU Utilization services were mod-inventory storage and nginx-okapi these 2 services were spiking at the beginning of the reindex and in the middle. No memory leaks. Database connection increased to 100 at the beginning and was equal to 700. DB CPU usage increased to 14%, this spike correlates with the changing of service CPU utilization and was caused by a high indexing data rate(about 110K operation per minute).

Open search metrics.

At the beginning of the reindex data rate was about 120K after an hour decreased to 25K, for the next 2 hours average data rate was ~ 25K, and for the last 5 hours decreased to 60. After the reindex data rate decreases to 60 the average indexing latency increase from 350 to 600 milliseconds. The data node's CPU utilization was about 60% at the beginning and correlated with the reindex data rate. CPU Utilization of the master node was changed from 5 to 27% during reindexing.

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Indexing Data Rate

Indexing latency

Master CPU Utilization (Average)

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Test #3 (cs00000intt_0002)

During the test on cs00000int tenant mod-inventory storage and nginx-okapi were spiking during the test. No memory leaks. Database connection increased by 50 and was equal to 650. DB CPU usage was also spiking from 100 to 20% during the test.

Open search metrics.

In the beginning, the reindex data rate was about 20K operations per minute and was spiking. After 2 hours reindex data rate decreased to 5K. Indexing latency increased was changing during the test from 3000 se. This spike correlates with the increasing of data nodes CPU utilization up to 16 % and next decreasing to 7%. CPU Utilization of the master node was changed from 5 to 27% during reindexing. The CPU on data nodes changes were not uniform and varied from 0 to 70%.

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Indexing Data Rate

Indexing latency

Master CPU Utilization (Average)

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Test #4 (In parallel: 3 tenants)

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Indexing Data Rate

Subrange of reindexing process from 22:30- 05:00. Graph was added to see detailed behavior aggregated on the graph above.

Indexing latency

Master CPU Utilization (Average)

CPU utilization percentage for all data nodes (Average).

Memory usage percentage for all data nodes (Average).

Appendix

Infrastructure

PTF-environment pcon

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instances of db.r6g.8xlarge database instances. Engine version 16.1
MSK ptf-kakfa-3
4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch fse cluster
- OpenSearch version 2.7;
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.4xlarge.search
  - Number of nodes - 6
  - EBS volume size (GiB) - 500
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s
- Dedicated master nodes
  - Instance type - r6g.large.search
  - Number of nodes - 3

Data structure

Module versions

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcon-pvt

mod-search	2	mod-search:3.2.0	8	2592	2480	2048	1440	512	1024	FALSE
mod-authtoken	1	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512	FALSE
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512	FALSE
mod-users	1	mod-users:19.3.1	2	1024	896	128	768	88	128	FALSE
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512	FALSE

Module	Task Def. Revision	Module Version	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcon-pvt
Mon Apr 29 08:13:56 UTC 2024
mod-remote-storage	1	mod-remote-storage:3.2.0	4920	4472	1024	3960	512	512	FALSE
mod-finance-storage	1	mod-finance-storage:8.6.0	1024	896	1024	700	88	128	FALSE
mod-ncip	1	mod-ncip:1.14.4	1024	896	128	768	88	128	FALSE
mod-agreements	1	mod-agreements:7.0.0	1592	1488	128	0	0	0	FALSE
mod-ebsconet	1	mod-ebsconet:2.2.0	1248	1024	128	700	128	256	FALSE
mod-consortia	1	mod-consortia:1.1.0	3072	2048	128	2048	512	1024	FALSE
mod-organizations	1	mod-organizations:1.9.0	1024	896	128	700	88	128	FALSE
mod-serials-management	1	mod-serials-management:1.0.0	2480	2312	128	1792	384	512	FALSE
mod-settings	1	mod-settings:1.0.3	1024	896	200	768	88	128	FALSE
mod-search	8	mod-search:3.2.0	2592	2480	2048	1440	512	1024	FALSE
edge-dematic	1	edge-dematic:2.2.0	1024	896	128	768	88	128	FALSE
mod-data-import	1	mod-data-import:3.1.0	2048	1844	256	1292	384	512	FALSE
mod-tags	1	mod-tags:2.2.0	1024	896	128	768	88	128	FALSE
mod-authtoken	1	mod-authtoken:2.15.1	1440	1152	512	922	88	128	FALSE
edge-courses	1	edge-courses:1.4.0	1024	896	128	768	88	128	FALSE
mod-inventory-update	1	mod-inventory-update:3.3.0	1024	896	128	768	88	128	FALSE
mod-notify	1	mod-notify:3.2.0	1024	896	128	768	88	128	FALSE
mod-configuration	1	mod-configuration:5.10.0	1024	896	128	768	88	128	FALSE
mod-orders-storage	1	mod-orders-storage:13.7.0	1024	896	512	700	88	128	FALSE
edge-caiasoft	1	edge-caiasoft:2.2.0	1024	896	128	768	88	128	FALSE
mod-login-saml	1	mod-login-saml:2.8.0	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	1	mod-erm-usage-harvester:4.5.0	1024	896	128	768	88	128	FALSE
mod-password-validator	1	mod-password-validator:3.2.0	1440	1298	128	768	384	512	FALSE
mod-licenses	1	mod-licenses:6.0.0	2480	2312	128	1792	384	512	FALSE
mod-gobi	1	mod-gobi:2.8.0	1024	896	128	700	88	128	FALSE
mod-bulk-operations	1	mod-bulk-operations:2.0.0	3072	2600	1024	1536	384	512	FALSE
mod-fqm-manager	1	mod-fqm-manager:2.0.1	3000	2600	128	2048	384	512	TRUE
edge-dcb	1	edge-dcb:1.1.0	1024	896	128	768	88	128	FALSE
mod-graphql	1	mod-graphql:1.12.1	1024	896	128	768	88	128	FALSE
mod-finance	1	mod-finance:4.9.0	1024	896	128	700	88	128	FALSE
mod-erm-usage	1	mod-erm-usage:4.7.0	1024	896	128	768	88	128	FALSE
mod-batch-print	1	mod-batch-print:1.1.0	1024	896	128	768	88	128	FALSE
mod-copycat	1	mod-copycat:1.6.0	1024	512	128	768	88	128	FALSE
mod-lists	1	mod-lists:2.0.0	3000	2600	128	2048	384	512	FALSE
mod-entities-links	1	mod-entities-links:3.0.0	2592	2480	400	1440	0	1024	FALSE
mod-permissions	2	mod-permissions:6.5.0	1684	1544	512	1024	384	512	FALSE
pub-edge	1	pub-edge:2023.06.14	1024	896	128	768	0	0	FALSE
mod-orders	1	mod-orders:12.8.0	2048	1440	1024	1024	384	512	FALSE
edge-patron	1	edge-patron:5.1.0	1024	896	256	768	88	128	FALSE
edge-ncip	1	edge-ncip:1.9.2	1024	896	128	768	88	128	FALSE
mod-users-bl	1	mod-users-bl:7.7.0	1440	1152	512	922	88	128	FALSE
mod-invoice	1	mod-invoice:5.8.0	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.1.0	4096	3690	2048	3076	384	512	FALSE
mod-user-import	1	mod-user-import:3.8.0	1024	896	128	768	88	128	FALSE
mod-sender	1	mod-sender:1.12.0	1024	896	128	768	88	128	FALSE
edge-oai-pmh	1	edge-oai-pmh:2.9.0	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	1	mod-data-export-worker:3.2.1	3072	2048	1024	2048	384	512	FALSE
mod-rtac	1	mod-rtac:3.6.0	1024	896	128	768	88	128	FALSE
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	1	mod-source-record-storage:5.8.0	5600	5000	2048	3500	384	512	FALSE
mod-calendar	1	mod-calendar:3.1.0	1024	896	128	768	88	128	FALSE
mod-event-config	1	mod-event-config:2.7.0	1024	896	128	768	88	128	FALSE
mod-courses	1	mod-courses:1.4.10	1024	896	128	768	88	128	FALSE
mod-circulation-item	1	mod-circulation-item:1.0.0	1024	896	128	0	0	0	FALSE
mod-inventory	1	mod-inventory:20.2.0	2880	2592	1024	1814	384	512	FALSE
mod-email	1	mod-email:1.17.0	1024	896	128	768	88	128	FALSE
mod-circulation	2	mod-circulation:24.2.1	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	1	mod-pubsub:2.13.0	1536	1440	1024	922	384	512	FALSE
mod-di-converter-storage	1	mod-di-converter-storage:2.2.0	1024	896	128	768	88	128	FALSE
edge-orders	1	edge-orders:3.0.0	1024	896	128	768	88	128	FALSE
edge-rtac	1	edge-rtac:2.7.1	1024	896	128	768	88	128	FALSE
mod-users	1	mod-users:19.3.1	1024	896	128	768	88	128	FALSE
mod-template-engine	1	mod-template-engine:1.20.0	1024	896	128	768	88	128	FALSE
mod-patron-blocks	1	mod-patron-blocks:1.10.0	1024	896	1024	768	88	128	FALSE
mod-audit	1	mod-audit:2.9.0	1024	896	128	768	88	128	FALSE
edge-fqm	1	edge-fqm:2.0.0	1024	896	128	768	88	128	FALSE
mod-source-record-manager	1	mod-source-record-manager:3.8.0	5600	5000	2048	3500	384	512	FALSE
nginx-edge	1	nginx-edge:2023.06.14	1024	896	128	0	0	0	FALSE
mod-quick-marc	1	mod-quick-marc:5.1.0	2288	2176	128	1664	384	512	FALSE
nginx-okapi	1	nginx-okapi:2023.06.14	1024	896	128	0	0	0	FALSE
okapi-b	1	okapi:5.3.0	1684	1440	1024	922	384	512	FALSE
mod-feesfines	1	mod-feesfines:19.1.0	1024	896	128	768	88	128	FALSE
mod-invoice-storage	1	mod-invoice-storage:5.8.0	1872	1536	1024	1024	384	512	FALSE
mod-service-interaction	1	mod-service-interaction:4.0.1	2048	1844	256	1290	384	512	FALSE
mod-dcb	1	mod-dcb:1.1.0	1024	896	128	768	88	128	FALSE
mod-patron	1	mod-patron:6.1.0	1024	896	128	768	88	128	FALSE
mod-data-export	1	mod-data-export:5.0.0	2048	1524	1024	0	0	0	FALSE
mod-oai-pmh	1	mod-oai-pmh:3.13.0	4096	3690	2048	3076	384	512	FALSE
edge-connexion	1	edge-connexion:1.2.0	1024	896	128	768	88	128	FALSE
mod-notes	1	mod-notes:5.2.0	1024	896	128	952	384	512	FALSE
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.0.0	1024	896	128	768	88	128	FALSE
mod-login	1	mod-login:7.11.0	1440	1298	1024	768	384	512	FALSE
mod-organizations-storage	1	mod-organizations-storage:4.7.0	1024	896	128	700	88	128	FALSE
mod-data-export-spring	1	mod-data-export-spring:3.2.0	2048	1844	256	1536	384	512	FALSE
pub-okapi	1	pub-okapi:2023.06.14	1024	896	128	768	0	0	FALSE
mod-eusage-reports	1	mod-eusage-reports:2.1.1	1024	896	128	768	88	128	FALSE

Methodology/Approach

Use consortium cluster for testing (qcon in our case).
Configure the environment in accordance with Infrastructure parameters and requirements that are in the ticket PERF-889: ECS Reindex for mod-search (classification browse)Closed
Reindex, get the results for indexing time, and index size. Use Steps for testing process#Reindex for details.
- Reindex process was started from the JMeter script using POST request /search/index/inventory/reindex with parameters

{ "recreateIndex": true,
"resourceName": "instance" }

Folio Development Teams

mod-search: Test Reindexing of Instances on consortium environment (Quesnelia)

Overview

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Resource utilization

Test #1 (cs00000int main tenant)

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Test #2 (cs00000intt_0001)

Memory Utilization

DB CPU Utilization

DB Connections

Test #3 (cs00000intt_0002)

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Test #4 (In parallel: 3 tenants)

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search metrics

Appendix

Infrastructure

Data structure

Module versions

Methodology/Approach

Related content