[Ramsons] List App test report for Contains any, Contains all, Not contains any, Not contains all Operators

Owned by Olga Kondratenko
Last updated: Dec 19, 2024


Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators. PERF-892 - Getting issue details... STATUS

Summary


  • Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.
  • Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager MODFQMMGR-501 - Getting issue details... STATUS but only for creating new lists while the test query is running.
  • For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
  • Approximately DB CPU usage is up to 98.5%.

Recommendations and Jiras

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

MODFQMMGR-506 - Getting issue details... STATUS

MODLISTS-149 - Getting issue details... STATUS

MODLISTS-150 - Getting issue details... STATUS

MODFQMMGR-522 - Getting issue details... STATUS

MODFQMMGR-501 - Getting issue details... STATUS

MODFQMMGR-505 - Getting issue details... STATUS

MODFQMMGR-510 - Getting issue details... STATUS

MODLISTS-147 - Getting issue details... STATUS

MODFQMMGR-518 - Getting issue details... STATUS

Results

List name Search from the number of recordsRecords found

 1 user

4 users

8 users

10 usersquery
Holdings all statistical code28,016,666500,00000:05:4000:17:4100:18:0400:24:32"holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]"
Holdings any statistical code28,016,6665,347,86602:03:17

04:56:23

05:01:3409:04:10holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not all statistical code28,016,6665,453,772

02:24:23 if success

  • [404 Not Found] during [GET] to [http://query/[id]/sortedIds?offset=3978000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"message":"Entity type with ID [id] not found","type":null,"code":null,"parameters":[{"key":"entityTypeId","value":"[id]"}]}]

    in 03:10:28, 03:22:46
05:08:4805:08:2404:31:33holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Holdings not any statistical code28,016,666105,90800:03:2100:18:3600:05:2900:06:38holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]
Items all statistical code27,957,839100,00000:06:0200:09:5500:07:0300:12:37items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items any statistical code27,957,8394,874,98202:36:3203:22:25

FAILED

SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...

FAILED in 03:42:33
[500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}]		items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not all statistical code27,957,8395,184,43302:01:1502:16:1204:58:38FAILED in 03:43:02
[500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}]		items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)]
Items not any statistical code27,957,839409,45100:08:3900:17:3300:19:3200:21:39items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)]
Users all departments250,0857,68600:00:1300:00:1200:00:1200:00:13users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e]
Users any departments250,08529,21600:00:1800:00:1900:00:2200:00:22users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not all departments250,08536,44500:00:2400:00:2400:00:2500:00:27users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
Users not any departments250,0857,22900:00:1200:00:1200:00:1200:00:13users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c]
POL all Acquisition unitsany0not able to find POL MODFQMMGR-506 - Getting issue details... STATUS ---vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL any Acquisition unitsany0not able to find POL MODFQMMGR-506 - Getting issue details... STATUS ---vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not all Acquisition unitsany0not able to find POL MODFQMMGR-506 - Getting issue details... STATUS ---vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
POL not any Acquisition unitsany0not able to find POL MODFQMMGR-506 - Getting issue details... STATUS ---vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041]
Instance all statistical code29,105,407

20,000 or "FAILED"

Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes. MODLISTS-147 - Getting issue details... STATUS FAILED in 02:35:45
[400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: []		FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...		03:56:59instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance any statistical code29,105,407"FAILED" MODFQMMGR-505 - Getting issue details... STATUS

Not able to create the list

MODFQMMGR-501 - Getting issue details... STATUS

--instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not all statistical code29,105,407"FAILED" MODFQMMGR-505 - Getting issue details... STATUS Not able to create the list MODFQMMGR-501 - Getting issue details... STATUS --instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]
Instance not any statistical code29,105,407

534,975 or "FAILED"

02:51:2902:47:14FAILED
SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from...		04:17:27instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC]


Instance CPU Utilization

Service Memory Utilization

mod-lists & mod-fqm-manager

Service CPU Utilization 

Service CPU Utilization without mod-fqm-manager

Service CPU Utilization with mod-fqm-manager

mod-lists



RDS CPU Utilization 



RDS Database Connections


Long-running queries: 

select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

  • mod_inventory_storage.instance = 29105407
  • mod_inventory_storage.item = 27957839
  • mod_inventory_storage.holdings_record = 28016666
  • mod_users.users = 260000

PTF-environment rcp1

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 2 instances of db.r6g.xlarge database, writer instance, and reader instance.

  • MSK - fse-tenant
    4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearch ptf-test cluster

    • OpenSearch version 2.13

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.2xlarge.search

      • Number of nodes - 4

      • EBS volume size (GiB) - 300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

Module versions

  • mod-fqm-manager -R/W split enabled

Cluster Resources - rcp1










Module

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

mod-configuration

1

mod-configuration:5.11.0-SNAPSHOT.355

2

1024

896

0

768

88

128

mod-orders-storage

1

mod-orders-storage:13.8.0-SNAPSHOT.243

2

1024

896

512

700

88

128

mod-fqm-manager

10

mod-fqm-manager:2.1.0-SNAPSHOT.1197

2

3000

2600

128

2048

384

512

mod-finance

1

mod-finance:5.0.0-SNAPSHOT.174

2

1024

896

0

700

88

128

mod-lists

5

mod-lists:2.1.0-SNAPSHOT.1303

2

1024

896

0

768

88

128

mod-entities-links

4

mod-entities-links:3.1.0-SNAPSHOT.113

2

2592

2480

400

1440

0

1024

mod-permissions

2

mod-permissions:6.6.0-SNAPSHOT.385

2

1684

1544

512

1024

384

512

mod-orders

1

mod-orders:12.9.0-SNAPSHOT.398

2

2048

1440

1024

1024

384

512

edge-patron

1

edge-patron:5.2.0-SNAPSHOT.174

2

1024

896

256

768

88

128

mod-invoice

1

mod-invoice:5.9.0-SNAPSHOT.205

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.2.0-SNAPSHOT.773

2

4096

3690

2048

3076

512

1024

mod-circulation-storage

1

mod-circulation-storage:17.3.0-SNAPSHOT.405

2

2880

2592

1536

1814

384

512

mod-source-record-storage

2

mod-source-record-storage:5.9.0-SNAPSHOT.405

2

5600

5000

2048

3500

384

512

mod-inventory

1

mod-inventory:20.3.0-SNAPSHOT.546

2

2880

2592

1024

1814

384

512

mod-users

1

mod-users:19.3.2-SNAPSHOT.696

2

1024

896

0

768

88

128

edge-fqm

1

edge-fqm:2.1.0-SNAPSHOT.100

2

1024

896

0

768

88

128

mod-source-record-manager

1

mod-source-record-manager:3.9.0-SNAPSHOT.353

2

5600

5000

2048

3500

384

512

nginx-edge

1

nginx-edge:2023.06.14

2

1024

896

0

0

0

0

nginx-okapi

1

nginx-okapi:2023.06.14

2

1024

896

0

0

0

0

okapi-b

1

okapi:5.3.0

3

1684

1440

1024

922

384

512

mod-invoice-storage

1

mod-invoice-storage:5.9.0-SNAPSHOT.138

2

1872

1536

1024

1024

384

512

mod-service-interaction

1

mod-service-interaction:4.1.0-SNAPSHOT.126

2

2048

1844

256

1290

384

512

mod-login

1

mod-login:7.12.0-SNAPSHOT.324

2

1440

1298

1024

768

384

512

mod-organizations-storage

1

mod-organizations-storage:4.8.0-SNAPSHOT.131

2

1024

896

0

700

88

128

pub-okapi

1

pub-okapi:2023.06.14

2

1024

896

0

768

0

0

fqm-manager revision

 mod-fqm-manager, rivision #10
"cpu": 128,
"memory": 3000,
"memoryReservation": 2600

{
"name": "mod-fqm-manager.query-retention-duration",
"value": "24h"
}

{
"name": "MAX_QUERY_SIZE",
"value": "30000000"
}

{
"name": "task.execution.pool.core-size",
"value": "9"
}


mod-list revision

 mod-list, rivision #5

"name": "mod-lists",
"cpu": 0,
"memory": 1024,
"memoryReservation": 896,

{
"name": "MAX_LIST_SIZE",
"value": "20000000"
}

{
"name": "FOLIO_SYSTEM_USER_ENABLED",
"value": "true"
},

{
"name": "mod-lists.general.refresh-query-timeout-minutes",
"value": "1440"
},


Methodology/Approach

To test Baseline for List refresh JMeter scripts were used.

Test preparation: 

  • New lists created with Contains any, Contains all, Not contains any, Not contains all Operators

Refresh list for 1, 4, 8, and 10 concurrent users tested