/
[Ramsons] List App test report for Contains any, Contains all, Not contains any, Not contains all Operators

[Ramsons] List App test report for Contains any, Contains all, Not contains any, Not contains all Operators

Dec 19, 2024

 

Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators. https://folio-org.atlassian.net/browse/PERF-892

Summary

 

  • Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.

  • Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-managerhttps://folio-org.atlassian.net/browse/MODFQMMGR-501 but only for creating new lists while the test query is running.

  • For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.

  • Approximately DB CPU usage is up to 98.5%.

Recommendations and Jiras

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

https://folio-org.atlassian.net/browse/MODFQMMGR-506

https://folio-org.atlassian.net/browse/MODLISTS-149

https://folio-org.atlassian.net/browse/MODLISTS-150

https://folio-org.atlassian.net/browse/MODFQMMGR-522

https://folio-org.atlassian.net/browse/MODFQMMGR-501

https://folio-org.atlassian.net/browse/MODFQMMGR-505

https://folio-org.atlassian.net/browse/MODFQMMGR-510

https://folio-org.atlassian.net/browse/MODLISTS-147

https://folio-org.atlassian.net/browse/MODFQMMGR-518

Results

List name 

Search from the number of records

Records found

 1 user

4 users

8 users

10 users

query

List name 

Search from the number of records

Records found

 1 user

4 users

8 users

10 users

query

Holdings all statistical code

28,016,666

500,000

00:05:40

00:17:41

00:18:04

00:24:32

"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"

Holdings any statistical code

28,016,666

5,347,866

02:03:17

04:56:23

05:01:34

09:04:10

holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Holdings not all statistical code

28,016,666

5,453,772

02:24:23 if success

  • [404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}]

    in 03:10:28, 03:22:46

05:08:48

05:08:24

04:31:33

holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Holdings not any statistical code

28,016,666

105,908

00:03:21

00:18:36

00:05:29

00:06:38

holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]

Items all statistical code

27,957,839

100,000

00:06:02

00:09:55

00:07:03

00:12:37

items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items any statistical code

27,957,839

4,874,982

02:36:32

03:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]

items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not all statistical code

27,957,839

5,184,433

02:01:15

02:16:12

04:58:38

FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]

items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]

Items not any statistical code

27,957,839

409,451

00:08:39

00:17:33

00:19:32

00:21:39

items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]

Users all departments

250,085

7,686

00:00:13

00:00:12

00:00:12

00:00:13

users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]

Users any departments

250,085

29,216

00:00:18

00:00:19

00:00:22

00:00:22

users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not all departments

250,085

36,445

00:00:24

00:00:24

00:00:25

00:00:27

users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

Users not any departments

250,085

7,229

00:00:12

00:00:12

00:00:12

00:00:13

users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]

POL all Acquisition units

any

0

not able to find POL https://folio-org.atlassian.net/browse/MODFQMMGR-506

-

-

-

vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL any Acquisition units

any

0

not able to find POL https://folio-org.atlassian.net/browse/MODFQMMGR-506

-

-

-

vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not all Acquisition units

any

0

not able to find POL https://folio-org.atlassian.net/browse/MODFQMMGR-506

-

-

-

vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

POL not any Acquisition units

any

0

not able to find POL https://folio-org.atlassian.net/browse/MODFQMMGR-506

-

-

-

vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]

Instance all statistical code

29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes. https://folio-org.atlassian.net/browse/MODLISTS-147

FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

03:56:59

instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance any statistical code

29,105,407

"FAILED"

https://folio-org.atlassian.net/browse/MODFQMMGR-505

Not able to create the list

https://folio-org.atlassian.net/browse/MODFQMMGR-501

-

-

instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not all statistical code

29,105,407

"FAILED"

https://folio-org.atlassian.net/browse/MODFQMMGR-505

Not able to create the list https://folio-org.atlassian.net/browse/MODFQMMGR-501

-

-

instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

Instance not any statistical code

29,105,407

534,975 or "FAILED"

02:51:29

02:47:14

FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...

04:17:27

instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]

 

Instance CPU Utilization

Service Memory Utilization

mod-lists & mod-fqm-manager

Service CPU Utilization 

Service CPU Utilization without mod-fqm-manager

Service CPU Utilization with mod-fqm-manager

mod-lists

 

 

RDS CPU Utilization 

 

 

RDS Database Connections

 

Long-running queries: 

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

  • mod_inventory_storage.instance = 29105407

  • mod_inventory_storage.item = 27957839

  • mod_inventory_storage.holdings_record = 28016666

  • mod_users.users = 260000

PTF-environment rcp1

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 2 instances of db.r6g.xlarge database, writer instance, and reader instance.

  • MSK - fse-tenant
    4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearch ptf-test cluster

    • OpenSearch version 2.13

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.2xlarge.search

      • Number of nodes - 4

      • EBS volume size (GiB) - 300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

Module versions

  • mod-fqm-manager -R/W split enabled

Cluster Resources - rcp1

 

 

 

 

 

 

 

 

 

Module

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

mod-configuration

1

mod-configuration:5.11.0-SNAPSHOT.355

2

1024

896

0

768

88

128

mod-orders-storage

1

mod-orders-storage:13.8.0-SNAPSHOT.243

2

1024

896

512

700

88

128

mod-fqm-manager

10

mod-fqm-manager:2.1.0-SNAPSHOT.1197

2

3000

2600

128

2048

384

512

mod-finance

1

mod-finance:5.0.0-SNAPSHOT.174

2

1024

896

0

700

88

128

mod-lists

5

mod-lists:2.1.0-SNAPSHOT.1303

2

1024

896

0

768

88

128