This document contains the results of testing List App refreshing of 200k records on multiple tenants with R/W split enabled (Poppy release). The goal of testing is to assess the performance of mod-lists with load spread across multiple tenants.

Ticket:

PERF-665: [List App] Test performance of refreshing a list with many records (with multiple tenants and R/W split enabled)ClosedPreview

• Tests showed the Lists App refresh of concurrent lists on 3 tenants are:

  • 1.5 mins for 3 concurrent lists refresh test (1 list refresh on each tenant);

  • 2.3 mins for 10 concurrent lists refresh test (3-4 lists refresh on each tenant).

• During load test for 30 lists (10 lists per tenant) some of the list refreshes failed. After the test end "isRefreshing" status remained "true" for each list. It was reset manually directly through the database. Details on the issue can be found here: Failed list refresh investigation.

• During the 10 lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions' CPU utilization exceeded 100% during 30 lists test.

• Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during the 30 lists test. In comparison with testing with R/W split disabled, RDS CPU utilization didn't decrease when DB R/W split was enabled.

• Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

PERF-732: Investigate failing list refresh during load testsClosedPreview

Query used in lists - "Item status != Available". List refresh result is about 200K records.

* Query used in lists - "Item status == Checked out". List refresh result is 200K records. Results are taken from previous test reports: [Poppy] List App with multiple workflows and R/W split disabled test report, [Orchid] List App test report

**Query used in lists - "Item status != Available". List refresh result is about 200K records.

***Details on the issue can be found at Failed list refresh investigation part

Open

During the 10-lists test CPU utilization reached 200% for mod-fqm-manager and 111% for mod-lists. Also, mod-permissions CPU utilization exceeded 100% during 30 lists test.

Open

Open

Memory utilization for mod-permissions increased from 48% to 76% during the tests. No memory leak is suspected for all the modules.

Maximum DB CPU utilization reached 83% (writer instance) and 99% (reader instance) during 30 lists test. 

In comparison with testing with R/W split disabled, RDS CPU utilization for the writer node decreased from 70% to 29% for 3 users test, from 95% to 68% for 10 users test. At the same time CPU load on the reader node is higher than on the writer.

Open

Results for multiple tenants and R/W split disabled testing (for comparison):

Details can be found here: [Poppy] List App with multiple tenants and R/W split disabled

Open

Writer DB node

Open

Reader DB node

Open

Writer DB node

Open

Reader DB node

Open

Long-running queries: 

In order to investigate the issue there were conducted addtional tests for single tenant (30 lists in parallel) and multiple tenants (10 lists on each of three tenants in parallel).

! For the second tenant there were no failed list refreshes during all the tests.

! After the test end "isRefreshing" status remained "true" for each of the failed lists. It was reset manually directly through the database.

"Reason for failure: org.springframework.dao.DataIntegrityViolationException: could not execute batch..."

"ERROR: insert or update on table "list_contents_11" violates foreign key constraint "fk_refresh_id""

"Key (refresh_id)=(2604f768-8365-4cd1-8d44-42ac2eb4b0cb) is not present in table "list_refresh_details""

Other log messages can be found in the faile attached:

Open

Reader node:

Writer node:

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

• MSK tenant
  
  

  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

1. Enable R/W split for mod-fqm-manager.

2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent lists (users).

3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp

4. Conduct tests with JMeter script for multiple tenants.