Table of Contents

Overview

...

Ticket:

...

• Load tests (multiple tenants, R/W split disabled) showed that ListApp refreshing duration is
  • 1.9 min for 3 concurrent lists test (1 list refresh on each tenant);
  • 3.4 min for 10 concurrent lists test (3-4 lists refresh on each tenants).
• During load test for 30 lists (10 lists per tenant) some of the list refreshes failed. After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database. Details on the issue can be found here: Failed list refresh investigation.
• Comparing to previous test results, Lists App refresh duration is about 20-30% higher with R/W split feature disabled.
• During the 10 lists refresh test CPU utilization reached 138% for mod-fqm-manager and 118% for mod-lists. In comparison with testing with R/W split enabled, CPU utilization for mod-fqm-manager is 62% lower for testing with R/W split disabled.
• Memory utilization for mod-fqm-manager increased from 37% to 49%, for mod-permissions - from 33% to 66% during the tests. No memory leak is suspected for all the modules.
• Maximum DB CPU utilization reached 99% (writer instance) during 30 lists test. In comparison with multiple tenants testing, RDS CPU utilization for writer instance is 20% lower for single tenant test.

Recommendations & Jiras 

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

...

**Results are taken from previous test report: [Poppy] List App with single tenant and R/W split enabled

***After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database. More details about failing refresh you can find at the link: Failed list refresh investigation

Service CPU Utilization

During the 10 lists test CPU utilization reached 138% for mod-fqm-manager and 118% for mod-lists. Also, mod-permissions CPU utilization exceeded 107% during 30 lists test.

...

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...

1. Disable R/W split for mod-fqm-manager.
2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent lists.
3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp
4. Conduct tests with JMeter script for multiple tenants.

...