Versions Compared

Old Version 5

changes.mady.by.user Hanna Tuhuzbayeva

Saved on

compared with

New Version 6

changes.mady.by.user Hanna Tuhuzbayeva

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Overview

This document contains the results of testing List App refreshing of 200k records on single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of mod-lists with load spread across multiple tenants.

Ticket:

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-718


Summary

  • Load tests showed that single tenant ListApp refreshing duration is:
    • 2.6 min for 1 user test;
    • 3.1 min for 3 user test;
    • 4.2 min for 5 user test;
    • 6.5 min for 10 user test.
    • Duration is below the maximum accepted value of 10 minutes.
  • Load tests showed that ListApp refreshing duration increases 2-3 times when conducted with the same user number but on 1 tenant instead of 3 tenants. For example, during 3 users test duration increased from 1.5 min to 3.1 min, for 10 users - from 2.3 min to 6.5 min.
  • During the 10 users test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.
  • Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 user test. 
  • No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Scenario

Data quantity

List App refresh

single tenant


1 user
3 users
5 users
10 users

Results

ListApp refresh, avg1 user3 users5 users10 users
Poppy, single tenant, R/W split enabled2.6 min3.1 min4.2 min6.5 min
Poppy, multiple tenants, R/W split enabled*
1.5 min
2.3 min

*Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split enabled

...

Reader DB node


Long-running queries: 

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'


delete from list_contents where list_id=$1 and refresh_id=$2


Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731


  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

Module
pcp1-pvt
Fri Oct 27 08:26:47 UTC 2023

Task Def. Revision

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mod-inventory-storage:27.0.01024096369020483076384512false
mod-users:19.2.0192102489612876888128false
nginx-okapi:2023.06.14821024896128000false
mod-circulation-storage:17.1.01022880259215361814384512false
okapi:5.1.193168414401024922384512false
mod-inventory:20.1.0922880259210241814384512false
mod-circulation:24.0.01022880259215361814384512false
pub-okapi:2023.06.1482102489612876800false
mod-fqm-manager:1.0.052102489612876888128true
mod-lists:1.0.052300026001282048384512false

Methodology

  1. Enable R/W split for mod-fqm-manager.
  2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent users.
  3. Prepare 200K item records for the query to return.
  4. Conduct tests with JMeter script (single tenant).