Overview

This document contains the results of testing List App refreshing of 200k records on multiple tenants with R/W split enabled (Poppy release). The goal of testing is to assess the performance of mod-lists with load spread across multiple tenants.

Ticket:

PERF-665 - Getting issue details... STATUS

Summary

• Tests showed the Lists App refresh of concurrent lists on 3 tenants are:
  • 1.5 mins for 3 concurrent lists refresh test (1 list refresh on each tenant);
  • 2.3 mins for 10 concurrent lists refresh test (3-4 lists refresh on each tenant).
• Load test for 30 lists (10 lists per tenant) failed due to DB overload (100% of refresh transactions failed). After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database.
• During the 10 lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions' CPU utilization exceeded 100% during 30 lists test.
• Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during the 30 lists test. In comparison with testing with R/W split disabled, RDS CPU utilization didn't decrease when DB R/W split was enabled.
• Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Results

* Query used in lists - "Item status == Checked out". List refresh result is 200K records. Results are taken from previous test reports: [Poppy] List App with multiple workflows and R/W split disabled test report, [Orchid] List App test report

**Query used in lists - "Item status != Available". List refresh result is about 200K records.

***After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database.

Instance CPU Utilization

Service CPU Utilization

During the 10-lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions CPU utilization exceeded 100% during 30 lists test.

Memory Utilization

Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

DB CPU Utilization

Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during 30 lists test. 

In comparison with testing with R/W split disabled, RDS CPU utilization for the writer node decreased from 70% to 29% for 3 users test, from 95% to 68% for 10 users test. At the same time CPU load on the reader node is higher than on the writer.

Results for multiple tenants and R/W split disabled testing (for comparison):

Details can be found here: [Poppy] List App with multiple tenants and R/W split disabled

DB Connections

DB Load

Writer DB node

Reader DB node

TOP SQL

Writer DB node

Reader DB node

Long-running queries: 

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'

delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology

1. Enable R/W split for mod-fqm-manager.
2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent lists (users).
3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp
4. Conduct tests with JMeter script for multiple tenants.