Versions Compared

Version Old Version 5 New Version 6
Changes made by

Olga Kondratenko

Olga Kondratenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

...

  • Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager
    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODFQMMGR-501
    but only for creating new lists while the test query is running.
  • For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
  • Approximately DB CPU usage is up to 98.5%.

...

  • is running.
  • For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
  • Approximately DB CPU usage is up to 98.5%.

Recommendations and Jiras

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODLISTS-149

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODLISTS-150

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-522

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-501

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-505

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-510

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODLISTS-147

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-518

Results

List name Search from the number of recordsRecords found

 1 user

4 users

8 users

10 usersquery
Holdings all statistical code28,016,666500,00000:05:4000:17:4100:18:0400:24:32"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"
Holdings any statistical code28,016,6665,347,86602:03:17

04:56:23

05:01:3409:04:10holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not all statistical code28,016,6665,453,772

02:24:23 if success

  • [404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}]

    in 03:10:28, 03:22:46
05:08:4805:08:2404:31:33holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not any statistical code28,016,666105,90800:03:2100:18:3600:05:2900:06:38holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Items all statistical code27,957,839100,00000:06:0200:09:5500:07:0300:12:37items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items any statistical code27,957,8394,874,98202:36:3203:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]		items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not all statistical code27,957,8395,184,43302:01:1502:16:1204:58:38FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]		items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not any statistical code27,957,839409,45100:08:3900:17:3300:19:3200:21:39items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]
Users all departments250,0857,68600:00:1300:00:1200:00:1200:00:13users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]
Users any departments250,08529,21600:00:1800:00:1900:00:2200:00:22users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not all departments250,08536,44500:00:2400:00:2400:00:2500:00:27users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not any departments250,0857,22900:00:1200:00:1200:00:1200:00:13users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
POL all Acquisition unitsany0not able to find POL
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506
---vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL any Acquisition unitsany0not able to find POL
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506
---vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not all Acquisition unitsany0not able to find POL
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506
---vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not any Acquisition unitsany0not able to find POL
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506
---vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
Instance all statistical code29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes.
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODLISTS-147
FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []		FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...		03:56:59instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance any statistical code29,105,407"FAILED"
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-505

Not able to create the list

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-501

--instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not all statistical code29,105,407"FAILED"
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-505
Not able to create the list
Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-501
--instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not any statistical code29,105,407

534,975 or "FAILED"

02:51:2902:47:14FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...		04:17:27instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]


Instance CPU Utilization

Service Memory Utilization

...

Records count :

  • mod_inventory_storage.item instance = 29105407
  • mod_inventory_storage.item = 27957839
  • mod_inventory_storage.item holdings_record = 28016666
  • mod_users.users = 

PTF -environment ncp5 

  • 12 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, one reader, and one writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731


  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...