Table of Contents

Overview

...

Table of Contents

Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-892

Summary

Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.
Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODFQMMGR-501
but only for creating new lists while the test query is running.
For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
Approximately DB CPU usage is up to 98.5%.

Recommendations and Jiras

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

Jira Legacy

key
server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc

...

Summary

...

MODFQMMGR-506

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODLISTS-149

Jira Legacy

key
server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc

...

Recommendations and Jiras

Results

...

1 user

...

4 users

...

8 users

...

04:56:23

...

02:24:23 if success

...

MODLISTS-150

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-522

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-510

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODLISTS-147

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-518

Results

List name	Search from the number of records	Records found	1 user	4 users	8 users	10 users	query
Holdings all statistical code	28,016,666	500,000	00:05:40	00:17:41	00:18:04	00:24:32	"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"
Holdings

not

any statistical code	28,016,666

105

5,347,

908

866

00

02:03:

21

17

00

04:

18

56:

36

23

00

05:

05

01:

29

34

00

09:

06

04:

38

10	holdings.statistical_code_names

does not contain

contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Items

Holdings not all statistical code

27

28,

957

016,

839

666

100

5,453,

000

772

00:06:

02

00

:

09

24:

5500:07:0300:12:37items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]Items any statistical code27,957,8394,874,98202:36:3203:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]Items not all statistical code27,957,8395,184,43302:01:1502:16:1204:58:38FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]Items not any statistical code27,957,839409,45100:08:3900:17:3300:19:3200:21:39items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]Users all departments250,0857,68600:00:1300:00:1200:00:1200:00:13users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]Users any departments250,08529,21600:00:1800:00:1900:00:2200:00:22users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]Users not all departments250,08536,44500:00:2400:00:2400:00:2500:00:27users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]Users not any departments250,0857,22900:00:1200:00:1200:00:1200:00:13users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]POL all Acquisition unitsany0not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

---vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]POL any Acquisition unitsany0not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

---vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]POL not all Acquisition unitsany0not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

---vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]POL not any Acquisition unitsany0not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

---vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]Instance all statistical code29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODLISTS-147

FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...03:56:59instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]Instance any statistical code29,105,407"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

--instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]Instance not all statistical code29,105,407"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

--instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]Instance not any statistical code29,105,407

534,975 or "FAILED"

02:51:2902:47:14FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...04:17:27instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Memory Utilization

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.

mod-lists

Service CPU Utilization

For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.

...

RDS CPU Utilization

RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%

RDS Database Connections

DB Load

Long-running queries:

Code Block

language	sql

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

mod_inventory_storage.item =
mod_inventory_storage.item =
mod_inventory_storage.item =
mod_users.users =

PTF -environment ncp5

...

2 database instances, one reader, and one writer

...

4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3

Modules memory and CPU parameters

Module
ncp5-pvt
Wed Aug 23 10:38:27 UTC 2023Task Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabledmod-remote-storage14mod-remote-storage:2.0.324920447210243960512512truemod-codex-mux3mod-codex-mux:2.13.0-SNAPSHOT.1082102489612876888128falsemod-finance-storage9mod-finance-storage:8.4.221024896102470088128falsemod-ncip9mod-ncip:1.13.12102489612876888128falsemod-agreements6mod-agreements:5.5.2215921488128968384512falsemod-ebsconet6mod-ebsconet:2.0.0212481024128700128256falsemod-organizations6mod-organizations:1.7.02102489612870088128falseedge-sip24edge-sip2:3.0.02102489612876888128falsemod-settings6mod-settings:1.0.021024896200000falsemod-data-import9mod-data-import:2.7.11204818442561292384512trueedge-dematic14edge-dematic:2.0.111024896128000falsemod-search8mod-search:2.0.1225922480204814405121024falsemod-inn-reach1mod-inn-reach:3.1.0-SNAPSHOT.4922304204851214405121024falsemod-tags6mod-tags:2.0.12102489612876888128falsemod-authtoken7mod-authtoken:2.13.021440115251292288128truemod-notify7mod-notify:3.0.02102489612876888128falsemod-inventory-update6mod-inventory-update:3.0.12102489612876888128falsemod-configuration9mod-configuration:5.9.12102489612876888128truemod-orders-storage6mod-orders-storage:13.5.02102489651270088128falseedge-caiasoft6edge-caiasoft:2.0.021024896128000falsemod-login-saml6mod-login-saml:2.6.22102489612876888128falsemod-erm-usage-harvester7mod-erm-usage-harvester:4.3.12102489612876888128falsemod-password-validator6mod-password-validator:3.0.0214401298128768384512falsemod-licenses6mod-licenses:4.3.12248023121281792384512falsemod-gobi6mod-gobi:2.6.02102489612870088128falsemod-bulk-operations17mod-bulk-operations:1.0.623072260010241536384512falsemod-fqm-manager19mod-fqm-manager:1.0.0-SNAPSHOT.8411300026001282048384512truemod-graphql6mod-graphql:1.11.02102489612876888128falsemod-finance6mod-finance:4.7.12102489612870088128falsemod-erm-usage7mod-erm-usage:4.5.32102489612876888128falsemod-copycat6mod-copycat:1.4.02102489612876888128falsemod-lists3-4-5mod-lists:1.0.0-SNAPSHOT.1130130002600128/5122048384512false/truemod-permissions20mod-permissions:6.3.22168415445121024384512truemod-entities-links7mod-entities-links:1.0.2225922480400144001024falsepub-edge7pub-edge:2023.06.142102489612876800falsemod-codex-inventory3mod-codex-inventory:2.4.0-SNAPSHOT.1392102489612876888128truemod-orders11mod-orders:12.6.822048144010241024384512falseedge-patron6edge-patron:4.11.02102489625676888128falseedge-ncip6edge-ncip:1.8.12102489612876888128falsemod-users-bl8mod-users-bl:7.5.021440115251292288128falsemod-inventory-storage11mod-inventory-storage:26.0.024096369020483076384512truemod-invoice6mod-invoice:5.6.521440115251292288128falsemod-user-import6mod-user-import:3.7.22102489612876888128falsemod-sender7mod-sender:1.10.02102489612876888128falseedge-oai-pmh9edge-oai-pmh:2.6.121512136010241440384512falsemod-data-export-worker10mod-data-export-worker:3.0.1323072280010242048384512falsemod-rtac8mod-rtac:3.5.02102489612876888128falsemod-task-list3mod-task-list:1.7.12102489612876888128falsemod-circulation-storage13mod-circulation-storage:16.0.0.1622880259215361814384512falsemod-calendar8mod-calendar:2.4.22102489612876888128falsemod-source-record-storage26mod-source-record-storage:5.6.725600500020483500384512truemod-event-config6mod-event-config:2.5.02102489612876888128falsemod-courses6mod-courses:1.4.72102489612876888128falsemod-inventory1mod-inventory:20.0.402880259210241814384512falsemod-email6mod-email:1.15.32102489612876888128falsemod-circulation7mod-circulation:23.5.6228802592153676888128falsemod-di-converter-storage6mod-di-converter-storage:2.0.52102489612876888128falsemod-pubsub10mod-pubsub:2.9.12153614401024922384512trueedge-orders6edge-orders:2.8.12102489612876888128falseedge-rtac6edge-rtac:2.6.02102489612876888128falsemod-template-engine6mod-template-engine:1.18.02102489612876888128falsemod-users8mod-users:19.1.12102489612876888128truemod-patron-blocks11mod-patron-blocks:1.8.021024896102476888128truemod-audit9mod-audit:2.7.02102489612876888128falsemod-source-record-manager18mod-source-record-manager:3.6.425600500020483500384512falsenginx-edge8nginx-edge:2023.06.1421024896128000falsemod-quick-marc6mod-quick-marc:3.0.01228821761281664384512falsenginx-okapi7nginx-okapi:2023.06.1421024896128000falseokapi-b8okapi:5.0.13168414401024922384512falsemod-feesfines9mod-feesfines:18.2.12102489612876888128truemod-invoice-storage6mod-invoice-storage:5.6.021872153610241024384512falsemod-codex-ekb3mod-codex-ekb:1.11.0-SNAPSHOT.1412102489612876888128falsemod-service-interaction6mod-service-interaction:2.2.22204818442561290384512falsemod-data-export6mod-data-export:4.7.111024896102476888128falsemod-patron7mod-patron:5.5.22102489612876888128falsemod-oai-pmh10mod-oai-pmh:3.11.324096369020483076384512trueedge-connexion6edge-connexion:1.0.62102489612876888128falsemod-kb-ebsco-java7mod-kb-ebsco-java:3.13.02102489612876888128falsemod-notes6mod-notes:5.0.121024896128952384512falsemod-data-export-spring8mod-data-export-spring:2.0.21204818442561536384512falsemod-organizations-storage7mod-organizations-storage:4.5.12102489612870088128falsemod-login6mod-login:7.9.02144012981024768384512falsepub-okapi7pub-okapi:2023.06.142102489612876800falsemod-data-import-converter-storage4mod-data-import-converter-storage:1.16.0-SNAPSHOT.1322102489612876888128falsemod-eusage-reports7mod-eusage-reports:1.3.02102489612876888128false

23 if success

[404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}]

in 03:10:28, 03:22:46

05:08:48

05:08:24

04:31:33

holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Holdings not any statistical code

28,016,666

105,908

00:03:21

00:18:36

00:05:29

00:06:38

holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Items all statistical code

27,957,839

100,000

00:06:02

00:09:55

00:07:03

00:12:37

items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items any statistical code

27,957,839

4,874,982

02:36:32

03:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]

items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not all statistical code

27,957,839

5,184,433

02:01:15

02:16:12

04:58:38

FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]

items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not any statistical code

27,957,839

409,451

00:08:39

00:17:33

00:19:32

00:21:39

items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]

Users all departments

250,085

7,686

00:00:13

00:00:12

00:00:13

users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]

Users any departments

250,085

29,216

00:00:18

00:00:19

00:00:22

users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not all departments

250,085

36,445

00:00:24

00:00:25

00:00:27

users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not any departments

250,085

7,229

00:00:12

00:00:13

users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

POL all Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL any Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not all Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not any Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

Instance all statistical code

29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODLISTS-147

FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

03:56:59

instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance any statistical code

29,105,407

"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

-

instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not all statistical code

29,105,407

"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

-

instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not any statistical code

29,105,407

534,975 or "FAILED"

02:51:29

02:47:14

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

04:17:27

instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance CPU Utilization

Image Added

Service Memory Utilization

Image Added

mod-lists & mod-fqm-manager

Image Added

Service CPU Utilization

Service CPU Utilization without mod-fqm-manager

Image Added

Service CPU Utilization with mod-fqm-managerImage Added

mod-lists

Image Added

RDS CPU Utilization

Image Added

RDS Database Connections

Image Added

Long-running queries:

Code Block

language	sql

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

mod_inventory_storage.instance = 29105407
mod_inventory_storage.item = 27957839
mod_inventory_storage.holdings_record = 28016666
mod_users.users = 260000

PTF-environment rcp1

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6g.xlarge database, writer instance, and reader instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch ptf-testcluster
- OpenSearch version 2.13
- Data nodes
  - Availability Zone(s) - 2-AZ without standby
  - Instance type - r6g.2xlarge.search
  - Number of nodes - 4
  - EBS volume size (GiB) - 300
  - Provisioned IOPS - 3000IOPS
  - Provisioned Throughput (MiB/s) - 250 MiB/s

Module versions

mod-fqm-manager -R/W split enabled

Cluster Resources - rcp1
Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-authtoken	1	mod-authtoken:2.16.0-SNAPSHOT.303	2	1440	1152	512	922	88	128
mod-configuration	1	mod-configuration:5.11.0-SNAPSHOT.355	2	1024	896	0	768	88	128
mod-orders-storage	1	mod-orders-storage:13.8.0-SNAPSHOT.243	2	1024	896	512	700	88	128
mod-fqm-manager	10	mod-fqm-manager:2.1.0-SNAPSHOT.1197	2	3000	2600	128	2048	384	512
mod-finance	1	mod-finance:5.0.0-SNAPSHOT.174	2	1024	896	0	700	88	128
mod-lists	5	mod-lists:2.1.0-SNAPSHOT.1303	2	1024	896	0	768	88	128
mod-entities-links	4	mod-entities-links:3.1.0-SNAPSHOT.113	2	2592	2480	400	1440	0	1024
mod-permissions	2	mod-permissions:6.6.0-SNAPSHOT.385	2	1684	1544	512	1024	384	512
mod-orders	1	mod-orders:12.9.0-SNAPSHOT.398	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.2.0-SNAPSHOT.174	2	1024	896	256	768	88	128
mod-invoice	1	mod-invoice:5.9.0-SNAPSHOT.205	2	1440	1152	512	922	88	128
mod-inventory-storage	4	mod-inventory-storage:27.2.0-SNAPSHOT.773	2	4096	3690	2048	3076	512	1024
mod-circulation-storage	1	mod-circulation-storage:17.3.0-SNAPSHOT.405	2	2880	2592	1536	1814	384	512
mod-source-record-storage	2	mod-source-record-storage:5.9.0-SNAPSHOT.405	2	5600	5000	2048	3500	384	512
mod-inventory	1	mod-inventory:20.3.0-SNAPSHOT.546	2	2880	2592	1024	1814	384	512
mod-users	1	mod-users:19.3.2-SNAPSHOT.696	2	1024	896	0	768	88	128
edge-fqm	1	edge-fqm:2.1.0-SNAPSHOT.100	2	1024	896	0	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.9.0-SNAPSHOT.353	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-invoice-storage	1	mod-invoice-storage:5.9.0-SNAPSHOT.138	2	1872	1536	1024	1024	384	512
mod-service-interaction	1	mod-service-interaction:4.1.0-SNAPSHOT.126	2	2048	1844	256	1290	384	512
mod-login	1	mod-login:7.12.0-SNAPSHOT.324	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.8.0-SNAPSHOT.131	2	1024	896	0	700	88	128
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	0	768	0	0

fqm-manager revision

Expand

title	mod-fqm-manager, rivision #10

"cpu": 128,
"memory": 3000,
"memoryReservation": 2600

{
"name": "mod-fqm-manager.query-retention-duration",
"value": "24h"
}

{
"name": "MAX_QUERY_SIZE",
"value": "30000000"
}

{
"name": "task.execution.pool.core-size",
"value": "9"
}

mod-list revision

Expand

title	mod-list, rivision #5

"name": "mod-lists",
"cpu": 0,
"memory": 1024,
"memoryReservation": 896,

{
"name": "MAX_LIST_SIZE",
"value": "20000000"
}

{
"name": "FOLIO_SYSTEM_USER_ENABLED",
"value": "true"
},

{
"name": "mod-lists.general.refresh-query-timeout-minutes",
"value": "1440"
},

Methodology/Approach

To test Baseline for List refresh JMeter scripts were used.

Test preparation:

Background circulation jobs were disabled.
200k items were checked out
10 lists were created with the query: (item_status == "Checked out") to be able to run a test for 10 concurrent users.New lists created with Contains any, Contains all, Not contains any, Not contains all Operators

Refresh list for 1, 24, 58, and 10 concurrent users tested

...

Version	Old Version 4	New Version Current
Changes made by	Olga Kondratenko	Olga Kondratenko
Saved on	Oct 10, 2024	Dec 19, 2024

Versions Compared

Key

Overview

Overview

Summary

Recommendations and Jiras

Summary

Recommendations and Jiras

Results

Results

Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections

DB Load

Appendix

Infrastructure

Instance CPU Utilization

Service Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections

Appendix

Infrastructure

Module versions

Methodology/Approach

Page Comparison

Versions Compared

Key

Overview

Overview

Summary

Recommendations and Jiras

Summary

Recommendations and Jiras

Results

Results

Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections

DB Load

Appendix

Infrastructure

Instance CPU Utilization

Service Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections

Appendix

Infrastructure

Module versions

Methodology/Approach