Table of Contents

Overview

...

Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODFQMMGR-501
but only for creating new lists while the test query is running.
For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 13 14 % for all other modules.
Approximately DB CPU usage is up to 98.5%.

...

List name

Search from the number of records

Records found

1 user

4 users

8 users

10 users

query

Holdings all statistical code

28,016,666

500,000

00:05:40

00:17:41

00:18:04

00:24:32

"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"

Holdings any statistical code

28,016,666

5,347,866

02:03:17

04:56:23

05:01:34

09:04:10

holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Holdings not all statistical code

28,016,666

5,453,772

02:24:23 if success

[404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}]

in 03:10:28, 03:22:46

05:08:48

05:08:24

04:31:33

holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Holdings not any statistical code

28,016,666

105,908

00:03:21

00:18:36

00:05:29

00:06:38

holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Items all statistical code

27,957,839

100,000

00:06:02

00:09:55

00:07:03

00:12:37

items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items any statistical code

27,957,839

4,874,982

02:36:32

03:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]

items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not all statistical code

27,957,839

5,184,433

02:01:15

02:16:12

04:58:38

FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]

items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not any statistical code

27,957,839

409,451

00:08:39

00:17:33

00:19:32

00:21:39

items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]

Users all departments

250,085

7,686

00:00:13

00:00:12

00:00:13

users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]

Users any departments

250,085

29,216

00:00:18

00:00:19

00:00:22

users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not all departments

250,085

36,445

00:00:24

00:00:25

00:00:27

users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not any departments

250,085

7,229

00:00:12

00:00:13

users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

POL all Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL any Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not all Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not any Acquisition units

any

0

not able to find POL

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-506

-

vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

Instance all statistical code

29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes.

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODLISTS-147

FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

03:56:59

instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance any statistical code

29,105,407

"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

-

instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not all statistical code

29,105,407

"FAILED"

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-505

Not able to create the list

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODFQMMGR-501

-

instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not any statistical code

29,105,407

534,975 or "FAILED"

02:51:29

02:47:14

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

04:17:27

instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Memory Utilization

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.

...

Instance CPU Utilization

Image Added

Service Memory Utilization

Image Added

mod-lists & mod-fqm-manager

Image Added

Service CPU Utilization

For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.Image Added

mod-fqm-manager

Image Added

mod-lists

Image Added

RDS CPU Utilization

RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%Image Added

RDS Database Connections

...

Image Added

Long-running queries:

Code Block

language	sql

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

...

mod_inventory_storage.item =
mod_inventory_storage.item =
mod_inventory_storage.item =
mod_users.users =

PTF -environment ncp5

12 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Version	Old Version 4	New Version 5
Changes made by	Olga Kondratenko	Olga Kondratenko
Saved on	Oct 10, 2024	Oct 10, 2024

Versions Compared

Key

Overview

Memory Utilization

Instance CPU Utilization

Service Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

Page Comparison

Versions Compared

Key

Overview

Memory Utilization

Instance CPU Utilization

Service Memory Utilization

Service CPU Utilization

RDS CPU Utilization

RDS Database Connections