Versions Compared

Version Old Version 6 New Version Current
Changes made by

Olga Kondratenko

Olga Kondratenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators.

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-892

Summary

...

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-501

...

Table of Contents


Overview

This document contains the results of testing List App refreshing of records in the Ramsons release for Contains any, Contains all, Not contains any, Not contains all Operators.

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-892

Summary


  • Some lists can’t be created at all, and even if they are, there's a high chance they’ll fail when trying to refresh.
  • Memory utilization increasing for mod-fgm-manager up to 80% during lists refresh then goes down in few minutes - OOM issue for mod-fqm-manager
    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODFQMMGR-501
    but only for creating new lists while the test query is running.
  • For mod-lists, CPU utilization was up to 237% for the test with 10 concurrent users and R/W split enabled. Average CPU usage did not exceed 14 % for all other modules.
  • Approximately DB CPU usage is up to 98.5%.

...

To have the possibility of refreshing lists for a large number of records, we need to increase QUERY RETENTION DURATION (mod-fqm-manager) and mod-lists.general.refresh-query-timeout-minutes to at least 10 hours.

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODFQMMGR-506

...

mod-lists & mod-fqm-manager

Service CPU Utilization 

Image Removed

Service CPU Utilization without mod-fqm-manager

Image Removed

mod-Image Added

Service CPU Utilization with mod-fqm-managerImage Added

mod-lists



RDS CPU Utilization 

...

Code Block
languagesql
select cast(array["instance.inst".id, ?] as varchar[]) from [tenant]_mod_fqm_manager.src_inventory_instance "instance.inst" left join [tenant]_mod_fqm_manager.src_inventory_mode_of_issuance "instance.mode_of_issuance" ON "instance.mode_of_issuance".id = "instance.inst".modeOfIssuanceId left join [tenant]_mod_fqm_manager.src_inventory_instance_type "instance.instance_type" ON "instance.instance_type".id = "instance.inst".instanceTypeId left join [tenant]_mod_fqm_manager.src_inventory_inst

Appendix

Infrastructure

Records count :

  • mod_inventory_storage.instance = 29105407
  • mod_inventory_storage.item = 27957839
  • mod_inventory_storage.holdings_record = 28016666
  • mod_users.users = 

PTF -environment ncp5 

...

2 database  instances, one reader, and one writer

...

  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Modules memory and CPU parameters

...

fqm_manager.src_inventory_inst


Appendix

Infrastructure

Records count :

  • mod_inventory_storage.instance = 29105407
  • mod_inventory_storage.item = 27957839
  • mod_inventory_storage.holdings_record = 28016666
  • mod_users.users = 260000

PTF-environment rcp1

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 2 instances of db.r6g.xlarge database, writer instance, and reader instance.

  • MSK - fse-tenant
    4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearch ptf-testcluster

    • OpenSearch version 2.13

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.2xlarge.search

      • Number of nodes - 4

      • EBS volume size (GiB) - 300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

Module versions

  • mod-fqm-manager -R/W split enabled

Cluster Resources - rcp1










Module

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

mod-configuration

1

mod-configuration:5.11.0-SNAPSHOT.355

2

1024

896

0

768

88

128

mod-orders-storage

1

mod-orders-storage:13.8.0-SNAPSHOT.243

2

1024

896

512

700

88

128

mod-fqm-manager

10

mod-fqm-manager:2.1.0-SNAPSHOT.1197

2

3000

2600

128

2048

384

512

mod-finance

1

mod-finance:5.0.0-SNAPSHOT.174

2

1024

896

0

700

88

128

mod-lists

5

mod-lists:2.1.0-SNAPSHOT.1303

2

1024

896

0

768

88

128

mod-entities-links

4

mod-entities-links:3.1.0-SNAPSHOT.113

2

2592

2480

400

1440

0

1024

mod-permissions

2

mod-permissions:6.6.0-SNAPSHOT.385

2

1684

1544

512

1024

384

512

mod-orders

1

mod-orders:12.9.0-SNAPSHOT.398

2

2048

1440

1024

1024

384

512

edge-patron

1

edge-patron:5.2.0-SNAPSHOT.174

2

1024

896

256

768

88

128

mod-invoice

1

mod-invoice:5.9.0-SNAPSHOT.205

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.2.0-SNAPSHOT.773

2

4096

3690

2048

3076

512

1024

mod-circulation-storage

1

mod-circulation-storage:17.3.0-SNAPSHOT.405

2

2880

2592

1536

1814

384

512

mod-source-record-storage

2

mod-source-record-storage:5.9.0-SNAPSHOT.405

2

5600

5000

2048

3500

384

512

mod-inventory

1

mod-inventory:20.3.0-SNAPSHOT.546

2

2880

2592

1024

1814

384

512

mod-users

1

mod-users:19.3.2-SNAPSHOT.696

2

1024

896

0

768

88

128

edge-fqm

1

edge-fqm:2.1.0-SNAPSHOT.100

2

1024

896

0

768

88

128

mod-source-record-manager

1

mod-source-record-manager:3.9.0-SNAPSHOT.353

2

5600

5000

2048

3500

384

512

nginx-edge

1

nginx-edge:2023.06.14

2

1024

896

0

0

0

0

nginx-okapi

1

nginx-okapi:2023.06.14

2

1024

896

0

0

0

0

okapi-b

1

okapi:5.3.0

3

1684

1440

1024

922

384

512

mod-invoice-storage

1

mod-invoice-storage:5.9.0-SNAPSHOT.138

2

1872

1536

1024

1024

384

512

mod-service-interaction

1

mod-service-interaction:4.1.0-SNAPSHOT.126

2

2048

1844

256

1290

384

512

mod-login

1

mod-login:7.12.0-SNAPSHOT.324

2

1440

1298

1024

768

384

512

mod-organizations-storage

1

mod-organizations-storage:4.8.0-SNAPSHOT.131

2

1024

896

0

700

88

128

pub-okapi

1

pub-okapi:2023.06.14

2

1024

896

0

768

0

0

fqm-manager revision

Expand
titlemod-fqm-manager, rivision #10
"cpu": 128,
"memory": 3000,
"memoryReservation": 2600

{
"name": "mod-fqm-manager.query-retention-duration",
"value": "24h"
}

{
"name": "MAX_QUERY_SIZE",
"value": "30000000"
}

{
"name": "task.execution.pool.core-size",
"value": "9"
}


mod-list revision

Expand
titlemod-list, rivision #5

"name": "mod-lists",
"cpu": 0,
"memory": 1024,
"memoryReservation": 896,

{
"name": "MAX_LIST_SIZE",
"value": "20000000"
}

{
"name": "FOLIO_SYSTEM_USER_ENABLED",
"value": "true"
},

{
"name": "mod-lists.general.refresh-query-timeout-minutes",
"value": "1440"
},



Methodology/Approach

To test Baseline for List refresh JMeter scripts were used.

Test preparation: 

  • Background circulation jobs were disabled.
  • 200k items were checked out
  • 10 lists were created with the query: (item_status == "Checked out") to be able to run a test for 10 concurrent users.New lists created with Contains any, Contains all, Not contains any, Not contains all Operators

Refresh list for 1, 24, 58, and 10 concurrent users tested

...