Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators. PERF-892 - Getting issue details... STATUS

Summary

Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.
Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager MODFQMMGR-501 - Getting issue details... STATUS but only for creating new lists while the test query is running.
For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
Approximately DB CPU usage is up to 98.5%.

Recommendations and Jiras

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

MODFQMMGR-506 - Getting issue details... STATUS

MODLISTS-149 - Getting issue details... STATUS

MODLISTS-150 - Getting issue details... STATUS

MODFQMMGR-522 - Getting issue details... STATUS

MODFQMMGR-501 - Getting issue details... STATUS

MODFQMMGR-505 - Getting issue details... STATUS

MODFQMMGR-510 - Getting issue details... STATUS

MODLISTS-147 - Getting issue details... STATUS

MODFQMMGR-518 - Getting issue details... STATUS

Results

List name	Search from the number of records	Records found	1 user	4 users	8 users	10 users	query
Holdings all statistical code	28,016,666	500,000	00:05:40	00:17:41	00:18:04	00:24:32	"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"
Holdings any statistical code	28,016,666	5,347,866	02:03:17	04:56:23	05:01:34	09:04:10	holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not all statistical code	28,016,666	5,453,772	02:24:23 if success [404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}] in 03:10:28, 03:22:46	05:08:48	05:08:24	04:31:33	holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not any statistical code	28,016,666	105,908	00:03:21	00:18:36	00:05:29	00:06:38	holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Items all statistical code	27,957,839	100,000	00:06:02	00:09:55	00:07:03	00:12:37	items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items any statistical code	27,957,839	4,874,982	02:36:32	03:22:25	FAILED SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...	FAILED in 03:42:33 [500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]	items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not all statistical code	27,957,839	5,184,433	02:01:15	02:16:12	04:58:38	FAILED in 03:43:02 [500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]	items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not any statistical code	27,957,839	409,451	00:08:39	00:17:33	00:19:32	00:21:39	items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]
Users all departments	250,085	7,686	00:00:13	00:00:12	00:00:12	00:00:13	users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]
Users any departments	250,085	29,216	00:00:18	00:00:19	00:00:22	00:00:22	users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not all departments	250,085	36,445	00:00:24	00:00:24	00:00:25	00:00:27	users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not any departments	250,085	7,229	00:00:12	00:00:12	00:00:12	00:00:13	users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
POL all Acquisition units	any	0	not able to find POL MODFQMMGR-506 - Getting issue details... STATUS	-	-	-	vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL any Acquisition units	any	0	not able to find POL MODFQMMGR-506 - Getting issue details... STATUS	-	-	-	vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not all Acquisition units	any	0	not able to find POL MODFQMMGR-506 - Getting issue details... STATUS	-	-	-	vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not any Acquisition units	any	0	not able to find POL MODFQMMGR-506 - Getting issue details... STATUS	-	-	-	vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
Instance all statistical code	29,105,407	20,000 or "FAILED"	Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes. MODLISTS-147 - Getting issue details... STATUS	FAILED in 02:35:45 [400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []	FAILED SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...	03:56:59	instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance any statistical code	29,105,407	"FAILED"	MODFQMMGR-505 - Getting issue details... STATUS	Not able to create the list MODFQMMGR-501 - Getting issue details... STATUS	-	-	instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not all statistical code	29,105,407	"FAILED"	MODFQMMGR-505 - Getting issue details... STATUS	Not able to create the list MODFQMMGR-501 - Getting issue details... STATUS	-	-	instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not any statistical code	29,105,407	534,975 or "FAILED"	02:51:29	02:47:14	FAILED SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...	04:17:27	instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]