/
[Poppy] List App with single tenant and R/W split enabled

[Poppy] List App with single tenant and R/W split enabled

Nov 09, 2023

Overview

This document contains the results of testing Lists App refreshing multiple concurrent lists of 200k records on single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of the Lists App modules (mod-lists, mod-fqm-manager) with load from across multiple users.

Ticket:https://folio-org.atlassian.net/browse/PERF-718

Summary

  • Load tests showed that single tenant ListApp refresh duration is:

    • 2.6 min for 1 refresh test;

    • 3.1 min for 3 concurrent refresh tests;

    • 4.2 min for 5 concurrent refresh tests;

    • 6.5 min for 10 concurrent refresh tests.

    • Duration is below the maximum accepted value of 10 minutes.

  • Tests showed that the same load (number of concurrent lists refreshes) on three tenants performs better than on one tenant. For example on one tenant, 3-concurrent list refreshes took 3.1 minutes on average, but on 3 tenants only 1.5 minutes. On one tenant 10 concurrent lists-refreshes took 6.5 minutes on average, but the same load on 3 tenants averaged 2.3 minutes.

  • During the 10 lists refresh test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists. In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for a single tenant test.

  • Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists-refresh test. In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.

  • No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Scenario

Data quantity

List App refresh

single tenant

 

1 list

3 concurrent lists

5 concurrent lists

10 concurrent lists

Results

Lists App refresh avg

1 list

3 lists

5 lists

10 lists

Poppy, single tenant, R/W split enabled

2.6 min

3.1 min

4.2 min

6.5 min

Poppy, multiple tenants, R/W split enabled*

 

1.5 min

 

2.3 min

Poppy, multiple tenants, R/W split disabled**

 

1.9 min

 

3.4 min

*Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split enabled

**Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split disabled

Service CPU Utilization

During the 10 lists refreshes test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.

In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for single tenant test.

Memory Utilization

No memory leak is suspected for all the modules.

DB CPU Utilization

Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists test. 

In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.

DB Connections

DB Load

TOP SQL

Writer DB node

Reader DB node

 

Long-running queries: 

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'

 

delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Module
pcp1-pvt
Fri Oct 27 08:26:47 UTC 2023

Task Def. Revision

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mod-inventory-storage:27.0.0

10

2

4096

3690

2048

3076

384

512

false

mod-users:19.2.0

19

2

1024

896

128

768

88

128

false

nginx-okapi:2023.06.14

8

2

1024

896

128

0

0

0

false

mod-circulation-storage:17.1.0

10

2

2880

2592

1536

1814

384

512

false

okapi:5.1.1

9

3

1684

1440

1024

922

384

512

false

mod-inventory:20.1.0

9

2

2880

2592

1024

1814

384

512

false

mod-circulation:24.0.0

10

2

2880

2592

1536

1814

384

512

false

pub-okapi:2023.06.14

8

2

1024

896

128

768

0

0

false

mod-fqm-manager:1.0.0

5

2

1024

896

128

768

88

128

true

mod-lists:1.0.0

5

2

3000

2600

128

2048

384

512

false

 

Methodology

  1. Enable R/W split for mod-fqm-manager.

  2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent users.

  3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp

  4. Conduct tests with JMeter script (single tenant).