Overview

The purpose of the document is to assess reindexing performance on a consortium environment. Calculate reindex time and size of index.

Recommendations & Jiras

Original ticket to test: PERF-635 - Getting issue details... STATUS
Additional info ElasticSearch Reindex Performance Recommendations

Test Summary

Reindexing process for consortium environment takes:
- 3 hours for 3 tenants in parallel (1.7M instances);
- 2 hours for central tenant reindexing (1.2M instances);
- 1 hour for secondary tenant reindexing (353K instances);
- 20 minutes for secondary tenant reindexing (202K instances).
Duration depends not only on instance number, but also its type (source). cs00000int_0001 tenant has a lot of shared instances (343K) but much less unshared ones (10K) compared to cs00000int_0002 (200K unshared). Data details can be found here: Datastructure
High CPU utilization is observed on nginx-okapi module - up to 413% during 3 tenants test.
CPU utilization for mod-inventory-storage reached 102% during test on central tenant. mod-search CPU utilization was about 13-27% during all the tests.
No memory leaks suspected.

Test Runs /Results

Test #	Instances number	Test Conditions reindexing on Poppy release, consortium environment	Duration *	Notes
1. 2023-11-28 09:20-12:20 UTC	1766108	In parallel: 3 tenants	3 hours	mod-search task count =8 mod-inventory-storage task count = 2 mod-okapi task count = 3 open search instance scaled up to r6g.4xlarge.search without configuration of number_of_replicas and refresh_interval values of ES/OpenSearch
2. 2023-11-29 08:50 - 10:50 UTC	1212927	Sequential: cs00000int	2 hours
3. 2023-11-29 14:05 - 15:05 UTC	353179	Sequential: cs00000int_0001	1 hour
4. 2023-11-2915:22 - 15:42 UTC	200002	Sequential: cs00000int_0002	20 min

*Duration depends not only on instance number, but also it's type (source). Data details can be found here: Datastructure

Indexing size

All the data from the tables below were capruted after each test was finished.

In parallel: 3 tenants
health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
green	open	pcon_instance_subject_cs00000int	oo6lG2KjRBm68SlF4cQf-A	4	2	851506	188613	3.4gb	1.1gb
green	open	pcon_contributor_cs00000int	D9izLpcOQWmqZFJhl6kyyA	4	2	855300	231525	1.8gb	685mb
green	open	pcon_authority_cs00000int	5CbWdgFrQSSBRlcNHPk7-A	4	2	0	0	2.4kb	832b
green	open	pcon_instance_cs00000int	6i5vkM6kRHOOu4ahRqESkA	4	2	1204190	1498	23.7gb	7.8gb

Results from get-request for reindex monitoring:

Tenant	Reindex id	Get request reindex
cs00000int	5dfca883-6236-438c-b4a4-bfd2274cdc0b	1212891
cs00000int_001	86a89e41-2858-4a9c-9e58-578ccf677413	353179
cs00000int_002	01ec9821-aca8-4a28-8c02-844807191ca9	200002
	SUM	1766072

health	status	index	uuid	pri	rep	docs.count	docs.deleted	store.size	pri.store.size
Sequential: cs00000int
green	open	pcon_instance_subject_cs00000int	XMqrUxTkTNKz4rIS1fi9Ug	4	2	862411	106300	3.2gb	1gb
green	open	pcon_contributor_cs00000int	TGA0HARfRLGXMDuFyQcmfg	4	2	835533	162792	1.5gb	560.8mb
green	open	pcon_authority_cs00000int	5CbWdgFrQSSBRlcNHPk7-A	4	2	0	0	2.4kb	832b
green	open	pcon_instance_cs00000int	R_Goc_w2T8CRnfvfbbsxHg	4	2	1212891	0	25.9gb	8.5gb
Sequential: cs00000int_0001
green	open	pcon_instance_subject_cs00000int	XMqrUxTkTNKz4rIS1fi9Ug	4	2	865366	83466	4.6gb	1.5gb
green	open	pcon_contributor_cs00000int	TGA0HARfRLGXMDuFyQcmfg	4	2	839478	90380	2.3gb	806mb
green	open	pcon_authority_cs00000int	5CbWdgFrQSSBRlcNHPk7-A	4	2	0	0	2.4kb	832b
green	open	pcon_instance_cs00000int	R_Goc_w2T8CRnfvfbbsxHg	4	2	1222891	161584	28.5gb	9.4gb
Sequential: cs00000int_0002
green	open	pcon_instance_subject_cs00000int	XMqrUxTkTNKz4rIS1fi9Ug	4	2	955467	126534	4.5gb	1.2gb
green	open	pcon_contributor_cs00000int	TGA0HARfRLGXMDuFyQcmfg	4	2	929045	189949	2.7gb	1gb
green	open	pcon_authority_cs00000int	5CbWdgFrQSSBRlcNHPk7-A	4	2	0	0	2.4kb	832b
green	open	pcon_instance_cs00000int	R_Goc_w2T8CRnfvfbbsxHg	4	2	1422889	161588	32.6gb	10.9gb

Results from get-request for reindex monitoring:

Tenant	Reindex id	Get request reindex
cs00000int	bb944cf4-b99f-4aa3-b13e-f5c92dc630ed	1212891
cs00000int_001	cf943d63-50db-4085-9629-783d7acdc67b	353179
cs00000int_002	c62e2662-6a21-47fd-9bb8-eea11364c2c1	200002
	SUM	1766072

Service CPU Utilization

Test #1 (3 tenants in parallel)

Maximum CPU utilization:

nginx-okapi - 413%

mod-inventory-storage - 95%

okapi - 73%

mod-search - 27%

Test #2 (cs00000int main tenant, sequential)

Maximum CPU utilization:

nginx-okapi - 332%

mod-inventory-storage - 102%

okapi - 63%

mod-search - 27%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Maximum CPU utilization:

nginx-okapi - 285%

mod-inventory-storage - 98%

okapi - 63%

mod-search - 13%

Memory Utilization

Test #1 (3 tenants in parallel)

Memory utilization:

mod-search - 38% → 50%

mod-inventory-storage - 11% → 31%

Test #2 (cs00000int main tenant, sequential)

Memory utilization:

mod-search - 37% → 50%

mod-inventory-storage - 13% → 21%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Memory utilization:

mod-search - 33% → 49%

mod-inventory-storage - 26% → 31%

DB CPU Utilization

Test #1 (3 tenants in parallel)

Maximum DB CPU utilization - 37%

Test #2 (cs00000int main tenant, sequential)

Maximum DB CPU utilization - 56%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Maximum DB CPU utilization - 36%

DB Connections

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequential)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Open Search CPU

Test #1 (3 tenants in parallel)

Maximum CPU utilization - 57%

Test #2 (cs00000int main tenant, sequential)

Maximum CPU utilization - 47%

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Maximum CPU utilization - 53%

Open Search Indexing Data Rate

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequental)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Open Search Indexing Latency

Test #1 (3 tenants in parallel)

Test #2 (cs00000int main tenant, sequential)

Test #3 and #4 (cs00000int_0001, cs00000int_0002 secondary tenants, sequential)

Appendix

Infrastructure

PTF-environment pcon

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6g.xlarge database instances, one reader and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Data structure

Tenant	Source	Instance number	Instances sum
cs00000int	FOLIO	115035	1212927
cs00000int	MARC	1097892	1212927
cs00000int_0001	CONSORTIUM-FOLIO	38712	353179
	CONSORTIUM-MARC	304467
	FOLIO	1000
	MARC	9000
cs00000int_0002	CONSORTIUM-MARC	4	200002
	FOLIO	30000
	MARC	169998

Module versions

Module pcon-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.0.0	2	4920	4471	1024	3960	512	512	false
mod-ncip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.1	2	1024	896	128	768	88	128	false
mod-finance-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128	false
mod-agreements	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:6.0.0	2	1592	1488	128	968	384	512	false
mod-ebsconet	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.1.0	2	1248	1024	128	700	128	256	false
mod-consortia	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.0.0	2	1024	896	128	768	88	128	false
edge-sip2	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.1.0	2	1024	896	128	768	88	128	false
mod-organizations	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.8.0	2	1024	896	128	768	88	128	false
mod-settings	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.1	2	1024	896	200	768	88	128	false
mod-search	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.0.3	2	2592	2480	2048	1440	512	1024	false
edge-dematic	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.1.0	1	1024	896	128	768	88	128	false
mod-data-import	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.0.0	1	2048	1844	256	1292	384	512	false
mod-tags	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.1.0	2	1024	896	128	768	88	128	false
mod-authtoken	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.0	2	1440	1152	512	922	88	128	false
mod-notify	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.1.0	2	1024	896	128	768	88	128	false
mod-inventory-update	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.2.1	2	1024	896	128	768	88	128	false
mod-configuration	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.2	2	1024	896	128	768	88	128	false
mod-orders-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.6.0	2	1024	896	512	768	88	128	false
edge-caiasoft	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.1.0	2	1024	896	128	768	88	128	false
mod-login-saml	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.7.0	2	1024	896	128	768	88	128	false
mod-gobi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.7.0	2	1024	896	128	768	88	128	false
mod-password-validator	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.1.0	2	1440	1298	128	768	384	512	false
mod-licenses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:5.0.0	2	2480	2312	128	1792	384	512	false
mod-bulk-operations	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:1.1.0	2	3072	2600	1024	1536	384	512	false
mod-graphql	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.0	2	1024	896	128	768	88	128	false
mod-finance	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.8.0	2	1024	896	128	768	88	128	false
mod-copycat	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.5.0	2	1024	896	128	768	88	128	false
mod-entities-links	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:2.0.0	2	2592	2480	400	1440	0	1024	false
mod-permissions	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.4.0	2	1684	1544	512	1024	384	512	false
pub-edge	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.14	2	1024	896	128	768	0	0	false
mod-orders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.7.0	2	2048	1440	1024	1024	384	512	false
edge-patron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.0.0	2	1024	896	256	768	88	128	false
edge-ncip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.0	2	1024	896	128	768	88	128	false
mod-users-bl	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	false
mod-inventory-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.0.0	2	4096	3690	2048	3076	384	512	false
mod-invoice	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.7.0	2	1440	1152	512	922	88	128	false
mod-user-import	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.0	2	1024	896	128	768	88	128	false
mod-sender	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.11.0	2	1024	896	128	768	88	128	false
edge-oai-pmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.7.0	2	1512	1360	1024	1440	384	512	false
mod-data-export-worker	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.1.0	2	3072	2800	1024	2048	384	512	false
mod-rtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.5.0	2	1024	896	128	768	88	128	false
mod-circulation-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.1.0	2	2880	2592	1536	1814	384	512	false
mod-source-record-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.7.0	2	5600	5000	2048	3500	384	512	false
mod-calendar	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:2.5.0	2	1024	896	128	768	88	128	false
mod-event-config	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.6.0	2	1024	896	128	768	88	128	false
mod-courses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.8	2	1024	896	128	768	88	128	false
mod-inventory	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.1.0	2	2880	2592	1024	1814	384	512	false
mod-email	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.16.0	2	1024	896	128	768	88	128	false
mod-di-converter-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.1.0	2	1024	896	128	768	88	128	false
mod-pubsub	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.11.0	2	1536	1440	1024	922	384	512	false
mod-circulation	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.0.0	2	2880	2592	1536	1814	384	512	false
edge-orders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:2.9.0	2	1024	896	128	768	88	128	false
edge-rtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.6.0	2	1024	896	128	768	88	128	false
mod-template-engine	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.19.1	2	1024	896	128	768	88	128	false
mod-users	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.2.0	2	1024	896	128	768	88	128	false
mod-patron-blocks	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	false
mod-audit	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.8.0	2	1024	896	128	768	88	128	false
mod-source-record-manager	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.7.0	2	5600	5000	2048	3500	384	512	false
nginx-edge	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.14	2	1024	896	128	0	0	0	false
mod-quick-marc	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.0.0	1	2288	2176	128	1664	384	512	false
nginx-okapi	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	false
okapi-b	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.0.0	2	1024	896	128	768	88	128	false
mod-invoice-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512	false
mod-service-interaction	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:3.0.0	2	2048	1844	256	1290	384	512	false
mod-patron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.0.0	2	1024	896	128	768	88	128	false
mod-data-export	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:4.8.0	1	1024	896	1024	768	88	128	false
mod-oai-pmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.12.0	2	4096	3690	2048	3076	384	512	false
edge-connexion	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.1.0	2	1024	896	128	768	88	128	false
mod-kb-ebsco-java	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	false
mod-notes	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.1.0	2	1024	896	128	952	384	512	false
mod-organizations-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.6.0	2	1024	896	128	768	88	128	false
mod-login	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.10.1	2	1440	1298	1024	768	384	512	false
mod-data-export-spring	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.0.0	1	2048	1844	256	1536	384	512	false
pub-okapi	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	128	768	0	0	false

Methodology/Approach

Use consortium cluster for testing (pcon in our case).
Configure the environment in accordance with Infrastructure parameters to the one that FSE commonly uses.
Run reindex, get the results for indexing time, index size. Use Steps for testing process#Reindex for details.
Compare results.

mod-search: Test Reindexing of Instances on consortium environment (Poppy)

Recommendations & Jiras

Test Summary

Test Runs /Results

Indexing size

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Open Search CPU

Open Search Indexing Data Rate

Open Search Indexing Latency

Appendix

Infrastructure

Data structure

Module versions

Methodology/Approach