Skip to end of banner
Go to start of banner

[Poppy] List App with single tenant and R/W split enabled

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Current »

Overview

This document contains the results of testing Lists App refreshing multiple concurrent lists of 200k records on single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of the Lists App modules (mod-lists, mod-fqm-manager) with load from across multiple users.

Ticket: PERF-718 - Getting issue details... STATUS

Summary

  • Load tests showed that single tenant ListApp refresh duration is:
    • 2.6 min for 1 refresh test;
    • 3.1 min for 3 concurrent refresh tests;
    • 4.2 min for 5 concurrent refresh tests;
    • 6.5 min for 10 concurrent refresh tests.
    • Duration is below the maximum accepted value of 10 minutes.
  • Tests showed that the same load (number of concurrent lists refreshes) on three tenants performs better than on one tenant. For example on one tenant, 3-concurrent list refreshes took 3.1 minutes on average, but on 3 tenants only 1.5 minutes. On one tenant 10 concurrent lists-refreshes took 6.5 minutes on average, but the same load on 3 tenants averaged 2.3 minutes.
  • During the 10 lists refresh test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists. In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for a single tenant test.
  • Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists-refresh test. In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.
  • No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Scenario

Data quantity

List App refresh

single tenant


1 list
3 concurrent lists
5 concurrent lists
10 concurrent lists

Results

Lists App refresh avg1 list3 lists5 lists10 lists
Poppy, single tenant, R/W split enabled2.6 min3.1 min4.2 min6.5 min
Poppy, multiple tenants, R/W split enabled*
1.5 min
2.3 min
Poppy, multiple tenants, R/W split disabled**
1.9 min
3.4 min

*Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split enabled

**Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split disabled

Service CPU Utilization

During the 10 lists refreshes test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.

In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for single tenant test.

Memory Utilization

No memory leak is suspected for all the modules.

DB CPU Utilization

Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists test. 

In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.

DB Connections

DB Load

TOP SQL

Writer DB node

Reader DB node


Long-running queries: 

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'


delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

Module
pcp1-pvt
Fri Oct 27 08:26:47 UTC 2023

Task Def. Revision

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mod-inventory-storage:27.0.01024096369020483076384512false
mod-users:19.2.0192102489612876888128false
nginx-okapi:2023.06.14821024896128000false
mod-circulation-storage:17.1.01022880259215361814384512false
okapi:5.1.193168414401024922384512false
mod-inventory:20.1.0922880259210241814384512false
mod-circulation:24.0.01022880259215361814384512false
pub-okapi:2023.06.1482102489612876800false
mod-fqm-manager:1.0.052102489612876888128true
mod-lists:1.0.052300026001282048384512false

Methodology

  1. Enable R/W split for mod-fqm-manager.
  2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent users.
  3. Prepare 200K item records for the query to return.
  4. Conduct tests with JMeter script (single tenant).
  • No labels