Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators. PERF-892 - Getting issue details... STATUS

Summary

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.
For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules.
Approximately DB CPU usage is up to 99.5%.

Recommendations and Jiras

Results

List name	Search from the number of records	Records found	1 user	4 users	8 users	10 users	query
Holdings all statistical code	28,016,666	500,000	00:05:40	00:17:41			"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"
Holdings any statistical code	28,016,666	5,347,866	02:03:17	05:47:55			holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not all statistical code	28,016,666	5,453,772	02:24:23 if success [404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}] in 03:10:28, 03:22:46	05:48:02			holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not any statistical code	28,016,666	105,908	00:03:21	00:18:36			holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Items all statistical code	27,957,839	100,000	00:06:02	00:09:55			items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items any statistical code	27,957,839	4,874,982	02:36:32	02:30:23			items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not all statistical code	27,957,839	5,184,433	02:01:15	06:07:59			items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not any statistical code	27,957,839	409,451	00:08:39	00:17:33			items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]
Users all departments	250,085	7,686	00:00:13	00:00:12			users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]
Users any departments	250,085	29,216	00:00:18	00:00:19			users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not all departments	250,085	36,445	00:00:24	00:00:24			users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not any departments	250,085	7,229	00:00:12	00:00:12			users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
POL all Acquisition units	any	0	not able to find POL	-	-	-	vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL any Acquisition units	any	0	not able to find POL	-	-	-	vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not all Acquisition units	any	0	not able to find POL	-	-	-	vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not any Acquisition units	any	0	not able to find POL	-	-	-	vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
Instance all statistical code	29,105,407	20,000 or "FAILED"	Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes. MODLISTS-147 - Getting issue details... STATUS	FAILED in 02:35:45 [400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []			instance.statistical_code_names contains all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance any statistical code	29,105,407	"FAILED"	MODFQMMGR-505 - Getting issue details... STATUS	Not able to create the list MODFQMMGR-501 - Getting issue details... STATUS	-	-	instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not all statistical code	29,105,407	"FAILED"	MODFQMMGR-505 - Getting issue details... STATUS	Not able to create the list MODFQMMGR-501 - Getting issue details... STATUS	-	-	instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not any statistical code	29,105,407	534,975 or "FAILED"	02:51:29	02:47:14			instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Memory Utilization

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.

mod-lists

Service CPU Utilization

For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.

mod-lists

RDS CPU Utilization

RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%

RDS Database Connections

DB Load

Long-running queries:

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

mod_inventory_storage.item = 22130108

PTF -environment ncp5

12 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module ncp5-pvt Wed Aug 23 10:38:27 UTC 2023	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage	14	mod-remote-storage:2.0.3	2	4920	4472	1024	3960	512	512	true
mod-codex-mux	3	mod-codex-mux:2.13.0-SNAPSHOT.108	2	1024	896	128	768	88	128	false
mod-finance-storage	9	mod-finance-storage:8.4.2	2	1024	896	1024	700	88	128	false
mod-ncip	9	mod-ncip:1.13.1	2	1024	896	128	768	88	128	false
mod-agreements	6	mod-agreements:5.5.2	2	1592	1488	128	968	384	512	false
mod-ebsconet	6	mod-ebsconet:2.0.0	2	1248	1024	128	700	128	256	false
mod-organizations	6	mod-organizations:1.7.0	2	1024	896	128	700	88	128	false
edge-sip2	4	edge-sip2:3.0.0	2	1024	896	128	768	88	128	false
mod-settings	6	mod-settings:1.0.0	2	1024	896	200	0	0	0	false
mod-data-import	9	mod-data-import:2.7.1	1	2048	1844	256	1292	384	512	true
edge-dematic	14	edge-dematic:2.0.1	1	1024	896	128	0	0	0	false
mod-search	8	mod-search:2.0.1	2	2592	2480	2048	1440	512	1024	false
mod-inn-reach	1	mod-inn-reach:3.1.0-SNAPSHOT.49	2	2304	2048	512	1440	512	1024	false
mod-tags	6	mod-tags:2.0.1	2	1024	896	128	768	88	128	false
mod-authtoken	7	mod-authtoken:2.13.0	2	1440	1152	512	922	88	128	true
mod-notify	7	mod-notify:3.0.0	2	1024	896	128	768	88	128	false
mod-inventory-update	6	mod-inventory-update:3.0.1	2	1024	896	128	768	88	128	false
mod-configuration	9	mod-configuration:5.9.1	2	1024	896	128	768	88	128	true
mod-orders-storage	6	mod-orders-storage:13.5.0	2	1024	896	512	700	88	128	false
edge-caiasoft	6	edge-caiasoft:2.0.0	2	1024	896	128	0	0	0	false
mod-login-saml	6	mod-login-saml:2.6.2	2	1024	896	128	768	88	128	false
mod-erm-usage-harvester	7	mod-erm-usage-harvester:4.3.1	2	1024	896	128	768	88	128	false
mod-password-validator	6	mod-password-validator:3.0.0	2	1440	1298	128	768	384	512	false
mod-licenses	6	mod-licenses:4.3.1	2	2480	2312	128	1792	384	512	false
mod-gobi	6	mod-gobi:2.6.0	2	1024	896	128	700	88	128	false
mod-bulk-operations	17	mod-bulk-operations:1.0.6	2	3072	2600	1024	1536	384	512	false
mod-fqm-manager	19	mod-fqm-manager:1.0.0-SNAPSHOT.841	1	3000	2600	128	2048	384	512	true
mod-graphql	6	mod-graphql:1.11.0	2	1024	896	128	768	88	128	false
mod-finance	6	mod-finance:4.7.1	2	1024	896	128	700	88	128	false
mod-erm-usage	7	mod-erm-usage:4.5.3	2	1024	896	128	768	88	128	false
mod-copycat	6	mod-copycat:1.4.0	2	1024	896	128	768	88	128	false
mod-lists	3-4-5	mod-lists:1.0.0-SNAPSHOT.1130	1	3000	2600	128/512	2048	384	512	false/true
mod-permissions	20	mod-permissions:6.3.2	2	1684	1544	512	1024	384	512	true
mod-entities-links	7	mod-entities-links:1.0.2	2	2592	2480	400	1440	0	1024	false
pub-edge	7	pub-edge:2023.06.14	2	1024	896	128	768	0	0	false
mod-codex-inventory	3	mod-codex-inventory:2.4.0-SNAPSHOT.139	2	1024	896	128	768	88	128	true
mod-orders	11	mod-orders:12.6.8	2	2048	1440	1024	1024	384	512	false
edge-patron	6	edge-patron:4.11.0	2	1024	896	256	768	88	128	false
edge-ncip	6	edge-ncip:1.8.1	2	1024	896	128	768	88	128	false
mod-users-bl	8	mod-users-bl:7.5.0	2	1440	1152	512	922	88	128	false
mod-inventory-storage	11	mod-inventory-storage:26.0.0	2	4096	3690	2048	3076	384	512	true
mod-invoice	6	mod-invoice:5.6.5	2	1440	1152	512	922	88	128	false
mod-user-import	6	mod-user-import:3.7.2	2	1024	896	128	768	88	128	false
mod-sender	7	mod-sender:1.10.0	2	1024	896	128	768	88	128	false
edge-oai-pmh	9	edge-oai-pmh:2.6.1	2	1512	1360	1024	1440	384	512	false
mod-data-export-worker	10	mod-data-export-worker:3.0.13	2	3072	2800	1024	2048	384	512	false
mod-rtac	8	mod-rtac:3.5.0	2	1024	896	128	768	88	128	false
mod-task-list	3	mod-task-list:1.7.1	2	1024	896	128	768	88	128	false
mod-circulation-storage	13	mod-circulation-storage:16.0.0.16	2	2880	2592	1536	1814	384	512	false
mod-calendar	8	mod-calendar:2.4.2	2	1024	896	128	768	88	128	false
mod-source-record-storage	26	mod-source-record-storage:5.6.7	2	5600	5000	2048	3500	384	512	true
mod-event-config	6	mod-event-config:2.5.0	2	1024	896	128	768	88	128	false
mod-courses	6	mod-courses:1.4.7	2	1024	896	128	768	88	128	false
mod-inventory	1	mod-inventory:20.0.4	0	2880	2592	1024	1814	384	512	false
mod-email	6	mod-email:1.15.3	2	1024	896	128	768	88	128	false
mod-circulation	7	mod-circulation:23.5.6	2	2880	2592	1536	768	88	128	false
mod-di-converter-storage	6	mod-di-converter-storage:2.0.5	2	1024	896	128	768	88	128	false
mod-pubsub	10	mod-pubsub:2.9.1	2	1536	1440	1024	922	384	512	true
edge-orders	6	edge-orders:2.8.1	2	1024	896	128	768	88	128	false
edge-rtac	6	edge-rtac:2.6.0	2	1024	896	128	768	88	128	false
mod-template-engine	6	mod-template-engine:1.18.0	2	1024	896	128	768	88	128	false
mod-users	8	mod-users:19.1.1	2	1024	896	128	768	88	128	true
mod-patron-blocks	11	mod-patron-blocks:1.8.0	2	1024	896	1024	768	88	128	true
mod-audit	9	mod-audit:2.7.0	2	1024	896	128	768	88	128	false
mod-source-record-manager	18	mod-source-record-manager:3.6.4	2	5600	5000	2048	3500	384	512	false
nginx-edge	8	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	false
mod-quick-marc	6	mod-quick-marc:3.0.0	1	2288	2176	128	1664	384	512	false
nginx-okapi	7	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	false
okapi-b	8	okapi:5.0.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	9	mod-feesfines:18.2.1	2	1024	896	128	768	88	128	true
mod-invoice-storage	6	mod-invoice-storage:5.6.0	2	1872	1536	1024	1024	384	512	false
mod-codex-ekb	3	mod-codex-ekb:1.11.0-SNAPSHOT.141	2	1024	896	128	768	88	128	false
mod-service-interaction	6	mod-service-interaction:2.2.2	2	2048	1844	256	1290	384	512	false
mod-data-export	6	mod-data-export:4.7.1	1	1024	896	1024	768	88	128	false
mod-patron	7	mod-patron:5.5.2	2	1024	896	128	768	88	128	false
mod-oai-pmh	10	mod-oai-pmh:3.11.3	2	4096	3690	2048	3076	384	512	true
edge-connexion	6	edge-connexion:1.0.6	2	1024	896	128	768	88	128	false
mod-kb-ebsco-java	7	mod-kb-ebsco-java:3.13.0	2	1024	896	128	768	88	128	false
mod-notes	6	mod-notes:5.0.1	2	1024	896	128	952	384	512	false
mod-data-export-spring	8	mod-data-export-spring:2.0.2	1	2048	1844	256	1536	384	512	false
mod-organizations-storage	7	mod-organizations-storage:4.5.1	2	1024	896	128	700	88	128	false
mod-login	6	mod-login:7.9.0	2	1440	1298	1024	768	384	512	false
pub-okapi	7	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	false
mod-data-import-converter-storage	4	mod-data-import-converter-storage:1.16.0-SNAPSHOT.132	2	1024	896	128	768	88	128	false
mod-eusage-reports	7	mod-eusage-reports:1.3.0	2	1024	896	128	768	88	128	false

Methodology/Approach

To test Baseline for List refresh JMeter scripts were used.

Test preparation:

Background circulation jobs were disabled.
200k items were checked out
10 lists were created with the query: (item_status == "Checked out") to be able to run a test for 10 concurrent users.

Refresh list for 1, 2, 5, and 10 concurrent users tested

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

[Ramsons] List App test report for Contains any, Contains all, Not contains any, Not contains all Operators