Skip to end of banner
Go to start of banner

[Poppy] List App with multiple tenants and R/W split enabled

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 28 Next »

Overview

This document contains the results of testing List App refreshing of 200k records on multiple tenants with R/W split enabled (Poppy release). The goal of testing is to assess the performance of mod-lists with load spread across multiple tenants.

Ticket:

PERF-665 - Getting issue details... STATUS

Summary

  • Tests showed the Lists App refresh of concurrent lists on 3 tenants are:
    • 1.5 mins for 3 concurrent lists refresh test (1 list refresh on each tenant);
    • 2.3 mins for 10 concurrent lists refresh test (3-4 lists refresh on each tenant).
  • Load test for 30 lists (10 lists per tenant) failed due to DB overload (100% of refresh transactions failed). After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database.
  • During the 10 lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions' CPU utilization exceeded 100% during 30 lists test.
  • Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during the 30 lists test. In comparison with testing with R/W split disabled, RDS CPU utilization didn't decrease when DB R/W split was enabled.
  • Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Scenario

Data quantity

List App refresh

multiple tenants




tenant 1 - 1 list

tenant 2 - 1 list

tenant 3 - 1 list

tenant 1 - 3 lists

tenant 2 - 3 lists

tenant 3 - 4 lists

tenant 1 - 10 lists

tenant 2 - 10 lists

tenant 3 - 10 lists


Results

TransactionDuration, avgReleaseTenantsNumber of listsR/W splitOther conditions

Lists App refresh

previous test results*

10 min 40 sec

[Orchid]1 tenant10 disabled
8.5 min[Poppy]1 tenant10 disabled
17.7 min[Poppy]1 tenant10 disabledTesting in parallel with DI and CICO
Lists App refresh

current test results**
1.5 min[Poppy]3 tenantsenabled
2.3 min[Poppy]3 tenants10 enabled
error[Poppy]3 tenants30 enabled100% of refresh transactions failed***

* Query used in lists - "Item status == Checked out". List refresh result is 200K records. Results are taken from previous test reports: [Poppy] List App with multiple workflows and R/W split disabled test report[Orchid] List App test report

**Query used in lists - "Item status != Available". List refresh result is about 200K records.

***After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database.

Instance CPU Utilization

Service CPU Utilization

During the 10-lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions CPU utilization exceeded 100% during 30 lists test.

Memory Utilization

Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

DB CPU Utilization

Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during 30 lists test. 

In comparison with testing with R/W split disabled, RDS CPU utilization for the writer node decreased from 70% to 29% for 3 users test, from 95% to 68% for 10 users test. At the same time CPU load on the reader node is higher than on the writer.

Results for multiple tenants and R/W split disabled testing (for comparison):

Details can be found here: [Poppy] List App with multiple tenants and R/W split disabled

DB Connections

DB Load

Writer DB node

Reader DB node

TOP SQL

Writer DB node

Reader DB node

Long-running queries: 

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'


delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instance, writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

Module
pcp1-pvt
Fri Oct 27 08:26:47 UTC 2023
Task Def. RevisionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-inventory-storage:27.0.01024096369020483076384512false
mod-users:19.2.0192102489612876888128false
nginx-okapi:2023.06.14821024896128000false
mod-circulation-storage:17.1.01022880259215361814384512false
okapi:5.1.193168414401024922384512false
mod-inventory:20.1.0922880259210241814384512false
mod-circulation:24.0.01022880259215361814384512false
pub-okapi:2023.06.1482102489612876800false
mod-fqm-manager:1.0.052102489612876888128true
mod-lists:1.0.052300026001282048384512false

Methodology

  1. Enable R/W split for mod-fqm-manager.
  2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent lists (users).
  3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp
  4. Conduct tests with JMeter script for multiple tenants.


  • No labels