Table of Contents |
---|
Overview
This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators.
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Summary
...
Table of Contents |
---|
Overview
This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators.
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Summary
- Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.
- Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager
but only for creating new lists while the test query is running.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFQMMGR-501 - For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 13 14 % for all other modules.
- Approximately DB CPU usage is up to 9998.5%.
Recommendations and Jiras
Results
...
1 user
...
4 users
...
8 users
...
02:24:23 if success
...
FAILED
SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from...
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
20,000 or "FAILED"
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Not able to create the list
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
534,975 or "FAILED"
...
Memory Utilization
Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.
mod-lists
Service CPU Utilization
For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.
...
RDS CPU Utilization
RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%
RDS Database Connections
DB Load
Long-running queries:
Code Block | ||
---|---|---|
| ||
select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst |
Appendix
Infrastructure
Records count :
- mod_inventory_storage.item = 22130108
PTF -environment ncp5
...
2 database instances, one reader, and one writer
...
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
...
To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Results
List name | Search from the number of records | Records found | 1 user | 4 users | 8 users | 10 users | query | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Holdings all statistical code | 28,016,666 | 500,000 | 00:05:40 | 00:17:41 | 00:18:04 | 00:24:32 | "holdings.statistical_code_names contains all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)]" | ||||||||||||||||
Holdings any statistical code | 28,016,666 | 5,347,866 | 02:03:17 | 04:56:23 | 05:01:34 | 09:04:10 | holdings.statistical_code_names contains any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)] | ||||||||||||||||
Holdings not all statistical code | 28,016,666 | 5,453,772 | 02:24:23 if success
| 05:08:48 | 05:08:24 | 04:31:33 | holdings.statistical_code_names does not contain all [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)] | ||||||||||||||||
Holdings not any statistical code | 28,016,666 | 105,908 | 00:03:21 | 00:18:36 | 00:05:29 | 00:06:38 | holdings.statistical_code_names does not contain any [PTF: PTF1 - PTF1, University of Chicago: ebooks - Books, electronic (ebooks)] | ||||||||||||||||
Items all statistical code | 27,957,839 | 100,000 | 00:06:02 | 00:09:55 | 00:07:03 | 00:12:37 | items.statistical_code_names contains all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)] | ||||||||||||||||
Items any statistical code | 27,957,839 | 4,874,982 | 02:36:32 | 03:22:25 | FAILED SQL [select cast(array["item.item".id, 'tenant'] as varchar[]) from... | FAILED in 03:42:33 [500 Internal Server Error] during [GET] to [http://query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds?offset=2004000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:55:52.355+00:00","status":500,"error":"Internal Server Error","path":"/query/73497f47-34fd-4eef-93db-4b858b20c6af/sortedIds"}] | items.statistical_code_names contains any [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)] | ||||||||||||||||
Items not all statistical code | 27,957,839 | 5,184,433 | 02:01:15 | 02:16:12 | 04:58:38 | FAILED in 03:43:02 [500 Internal Server Error] during [GET] to [http://query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds?offset=3989000&limit=1000] [QueryClient#getSortedIds(UUID,Integer,Integer)]: [{"timestamp":"2024-10-09T13:56:23.683+00:00","status":500,"error":"Internal Server Error","path":"/query/22bdc696-558e-47f5-9661-f940369359e2/sortedIds"}] | items.statistical_code_names does not contain all [University of Chicago: books - Books, print (books), University of Chicago: mfilm - Microfilm (mfilm)] | ||||||||||||||||
Items not any statistical code | 27,957,839 | 409,451 | 00:08:39 | 00:17:33 | 00:19:32 | 00:21:39 | items.statistical_code_names does not contain any [University of Chicago: mfilm - Microfilm (mfilm), University of Chicago: books - Books, print (books)] | ||||||||||||||||
Users all departments | 250,085 | 7,686 | 00:00:13 | 00:00:12 | 00:00:12 | 00:00:13 | users.departments contains all [9af255eb-b87c-4045-9407-d9c324bf0e8c, b1318aeb-0273-42b3-ba57-f1ee5d5cc59e] | ||||||||||||||||
Users any departments | 250,085 | 29,216 | 00:00:18 | 00:00:19 | 00:00:22 | 00:00:22 | users.departments contains any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c] | ||||||||||||||||
Users not all departments | 250,085 | 36,445 | 00:00:24 | 00:00:24 | 00:00:25 | 00:00:27 | users.departments does not contain all [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, 2eceb91f-6230-4fdb-9d22-8b34216ab800, 9af255eb-b87c-4045-9407-d9c324bf0e8c] | ||||||||||||||||
Users not any departments | 250,085 | 7,229 | 00:00:12 | 00:00:12 | 00:00:12 | 00:00:13 | users.departments does not contain any [b1318aeb-0273-42b3-ba57-f1ee5d5cc59e, fa6b405f-311e-493c-be6f-41610552742c, 9af255eb-b87c-4045-9407-d9c324bf0e8c] | ||||||||||||||||
POL all Acquisition units | any | 0 | not able to find POL
| - | - | - | vendor_organization.accounts[*]->acq_unit_ids contains all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041] | ||||||||||||||||
POL any Acquisition units | any | 0 | not able to find POL
| - | - | - | vendor_organization.accounts[*]->acq_unit_ids contains any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041] | ||||||||||||||||
POL not all Acquisition units | any | 0 | not able to find POL
| - | - | - | vendor_organization.accounts[*]->acq_unit_ids does not contain all [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041] | ||||||||||||||||
POL not any Acquisition units | any | 0 | not able to find POL
| - | - | - | vendor_organization.accounts[*]->acq_unit_ids does not contain any [d7515bef-206f-4f23-be80-563f9bca866b, 0ebb1f7d-983f-3026-8a4c-5318e0ebc041] | ||||||||||||||||
Instance all statistical code | 29,105,407 | 20,000 or "FAILED" | Condition with org.folio.list.services.refresh.ListRefreshService was not fulfilled within 1 hours 30 minutes.
| FAILED in 02:35:45 [400 Bad Request] during [GET] to [http://query/b793e898-9a17-4b28-bcd5-38fd89f55ba1] [QueryClient#getQuery(UUID)]: [] | FAILED SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from... | 03:56:59 | instance.statistical_code_names contains all [University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC] | ||||||||||||||||
Instance any statistical code | 29,105,407 | "FAILED" |
| Not able to create the list
| - | - | instance.statistical_code_names contains any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC] | ||||||||||||||||
Instance not all statistical code | 29,105,407 | "FAILED" |
| Not able to create the list
| - | - | instance.statistical_code_names not contain all [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC] | ||||||||||||||||
Instance not any statistical code | 29,105,407 | 534,975 or "FAILED" | 02:51:29 | 02:47:14 | FAILED SQL [select cast(array["instance.inst".id, 'tenant'] as varchar[]) from... | 04:17:27 | instance.statistical_code_names not contain any [PTF: PTF1 - PTF1, PTF: PTF3 - PTF3, University of Chicago: books - Books, print (books), RECM (Record management): XOCLC - Do not share with OCLC] |
Instance CPU Utilization
Service Memory Utilization
mod-lists & mod-fqm-manager
Service CPU Utilization
Service CPU Utilization without mod-fqm-manager
RDS CPU Utilization
RDS Database Connections
Long-running queries:
Code Block | ||
---|---|---|
| ||
select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst |
Appendix
Infrastructure
Records count :
- mod_inventory_storage.instance = 29105407
- mod_inventory_storage.item = 27957839
- mod_inventory_storage.holdings_record = 28016666
- mod_users.users = 260000
PTF-environment rcp1
10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6g.xlarge database, writer instance, and reader instance.
MSK - fse-tenant
4 kafka.m7g.xlarge brokers in 2 zonesApache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
OpenSearch ptf-testcluster
OpenSearch version 2.13
Data nodes
Availability Zone(s) - 2-AZ without standby
Instance type - r6g.2xlarge.search
Number of nodes - 4
EBS volume size (GiB) - 300
Provisioned IOPS - 3000IOPS
Provisioned Throughput (MiB/s) - 250 MiB/s
Module versions
mod-fqm-manager -R/W split enabled
Cluster Resources - rcp1 | |||||||||
Module | Task Definition Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft Limit | CPU Units | Xmx | Metaspace Size | Max Metaspace Size |
mod-authtoken | 1 | mod-authtoken:2.16.0-SNAPSHOT.303 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-configuration | 1 | mod-configuration:5.11.0-SNAPSHOT.355 | 2 | 1024 | 896 | 0 | 768 | 88 | 128 |
mod-orders-storage | 1 | mod-orders-storage:13.8.0-SNAPSHOT.243 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 |
mod-fqm-manager | 10 | mod-fqm-manager:2.1.0-SNAPSHOT.1197 | 2 | 3000 | 2600 | 128 | 2048 | 384 | 512 |
mod-finance | 1 | mod-finance:5.0.0-SNAPSHOT.174 | 2 | 1024 | 896 | 0 | 700 | 88 | 128 |
mod-lists | 5 | mod-lists:2.1.0-SNAPSHOT.1303 | 2 | 1024 | 896 | 0 | 768 | 88 | 128 |
mod-entities-links | 4 | mod-entities-links:3.1.0-SNAPSHOT.113 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 |
mod-permissions | 2 | mod-permissions:6.6.0-SNAPSHOT.385 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 |
mod-orders | 1 | mod-orders:12.9.0-SNAPSHOT.398 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 |
edge-patron | 1 | edge-patron:5.2.0-SNAPSHOT.174 | 2 | 1024 | 896 | 256 | 768 | 88 | 128 |
mod-invoice | 1 | mod-invoice:5.9.0-SNAPSHOT.205 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-inventory-storage | 4 | mod-inventory-storage:27.2.0-SNAPSHOT.773 | 2 | 4096 | 3690 | 2048 | 3076 | 512 | 1024 |
mod-circulation-storage | 1 | mod-circulation-storage:17.3.0-SNAPSHOT.405 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
mod-source-record-storage | 2 | mod-source-record-storage:5.9.0-SNAPSHOT.405 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
mod-inventory | 1 | mod-inventory:20.3.0-SNAPSHOT.546 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
mod-users | 1 | mod-users:19.3.2-SNAPSHOT.696 | 2 | 1024 | 896 | 0 | 768 | 88 | 128 |
edge-fqm | 1 | edge-fqm:2.1.0-SNAPSHOT.100 | 2 | 1024 | 896 | 0 | 768 | 88 | 128 |
mod-source-record-manager | 1 | mod-source-record-manager:3.9.0-SNAPSHOT.353 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
nginx-edge | 1 | nginx-edge:2023.06.14 | 2 | 1024 | 896 | 0 | 0 | 0 | 0 |
nginx-okapi | 1 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 0 | 0 | 0 | 0 |
okapi-b | 1 | okapi:5.3.0 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
mod-invoice-storage | 1 | mod-invoice-storage:5.9.0-SNAPSHOT.138 | 2 | 1872 | 1536 | 1024 | 1024 | 384 | 512 |
mod-service-interaction | 1 | mod-service-interaction:4.1.0-SNAPSHOT.126 | 2 | 2048 | 1844 | 256 | 1290 | 384 | 512 |
mod-login | 1 | mod-login:7.12.0-SNAPSHOT.324 | 2 | 1440 | 1298 | 1024 | 768 | 384 | 512 |
mod-organizations-storage | 1 | mod-organizations-storage:4.8.0-SNAPSHOT.131 | 2 | 1024 | 896 | 0 | 700 | 88 | 128 |
pub-okapi | 1 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 0 | 768 | 0 | 0 |
fqm-manager revision
Expand | ||
---|---|---|
| ||
{ { { |
mod-list revision
Expand | ||
---|---|---|
| ||
"name": "mod-lists", { { { |
Methodology/Approach
To test Baseline for List refresh JMeter scripts were used.
Test preparation:
- Background circulation jobs were disabled.
- 200k items were checked out
- 10 lists were created with the query: (item_status == "Checked out") to be able to run a test for 10 concurrent users.New lists created with Contains any, Contains all, Not contains any, Not contains all Operators
Refresh list for 1, 24, 58, and 10 concurrent users tested
...