Table of Contents

Overview

This document contains the results of testing List Lists App refreshing multiple concurrent lists of 200k records on single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of the Lists App modules (mod-lists, mod-fqm-manager) with load spread from across multiple tenantsusers.

Ticket:

Summary

• Load tests showed that single tenant ListApp refresh duration is:
  • 2.6 min for 1 refresh test;
  • 3.1 min for 3 concurrent refresh tests;
  • 4.2 min for 5 concurrent refresh tests;
  • 6.5 min for 10 concurrent refresh tests.
  • Duration is below the maximum accepted value of 10 minutes.
• Tests showed that the same load (number of concurrent lists refreshes) on three tenants performs better than on one tenant. For example on one tenant, 3-concurrent list refreshes took 3.1 minutes on average, but on 3 tenants only 1.5 minutes. On one tenant 10 concurrent lists-refreshes took 6.5 minutes on average, but the same load on 3 tenants averaged 2.3 minutes.
• During the 10 lists refresh test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists. In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for a single tenant test.
• Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists-refresh test. In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.
• No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Results

...

Lists App refresh avg	1 list	3 lists	5 lists	10 lists
Poppy, single tenant, R/W split enabled

1 user

3 users

5 users

10 usersListApp refresh, avg

	2.6 min	3.1 min	4.2 min	6.5 min
Poppy, multiple tenants, R/W split enabled*		1.5 min		2.3 min
Poppy, multiple tenants, R/W split disabled**		1.9 min		3.4 min

*Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split enabled

**Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split disabled

Service CPU Utilization

During the 10 lists refreshes test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.

In comparison with multiple tenants testing, CPU utilization for mod-fqm-manager is about 2 times lower for single tenant test.

Image Modified

Memory Utilization

No memory leak is suspected for all the modules.

Image Modified

DB CPU Utilization

Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 lists test. 

In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.

Image Modified

DB Connections

DB Load

TOP SQL

Writer DB node

Reader DB node

Long-running queries: 

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology

1. Enable R/W split for mod-fqm-manager.
2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent users.
3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp
4. Conduct tests with JMeter script (single tenant).