Overview

This document contains the results of testing List App refreshing of 200k records on single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of mod-lists with load spread across multiple tenants.

Ticket: PERF-718 - Getting issue details... STATUS

Summary

Load tests showed that single tenant ListApp refreshing duration is:
- 2.6 min for 1 user test;
- 3.1 min for 3 user test;
- 4.2 min for 5 user test;
- 6.5 min for 10 user test.
- Duration is below the maximum accepted value of 10 minutes.
Load tests showed that ListApp refreshing duration increases 2-3 times when conducted with the same user number but on 1 tenant instead of 3 tenants. For example, during 3 users test duration increased from 1.5 min to 3.1 min, for 10 users - from 2.3 min to 6.5 min.
During the 10 users test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.
Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 user test.
No memory leak is suspected for all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is about 200K records.

Scenario	Data quantity
List App refresh single tenant	1 user
	3 users
	5 users
	10 users

Results

ListApp refresh, avg	1 user	3 users	5 users	10 users
Poppy, single tenant, R/W split enabled	2.6 min	3.1 min	4.2 min	6.5 min
Poppy, multiple tenants, R/W split enabled*		1.5 min		2.3 min

*Results are taken from previous test report: [Poppy] List App with multiple tenants and R/W split enabled

Service CPU Utilization

During the 10 users test CPU utilization reached 89% for mod-fqm-manager and 101% for mod-lists.

Memory Utilization

No memory leak is suspected for all the modules.

DB CPU Utilization

Maximum DB CPU utilization reached 63% (writer instance) and 99% (reader instance) during 10 user test.

DB Connections

DB Load

TOP SQL

Writer DB node

Reader DB node

Long-running queries:

select id from [tenant]_mod_fqm_manager.drv_item_callnumber_location where lower(cast(item_status as varchar)) <> lower($1)
parameters: $1 = 'Available'

delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module pcp1-pvt Fri Oct 27 08:26:47 UTC 2023	Task Def. Revision	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-inventory-storage:27.0.0	10	2	4096	3690	2048	3076	384	512	false
mod-users:19.2.0	19	2	1024	896	128	768	88	128	false
nginx-okapi:2023.06.14	8	2	1024	896	128	0	0	0	false
mod-circulation-storage:17.1.0	10	2	2880	2592	1536	1814	384	512	false
okapi:5.1.1	9	3	1684	1440	1024	922	384	512	false
mod-inventory:20.1.0	9	2	2880	2592	1024	1814	384	512	false
mod-circulation:24.0.0	10	2	2880	2592	1536	1814	384	512	false
pub-okapi:2023.06.14	8	2	1024	896	128	768	0	0	false
mod-fqm-manager:1.0.0	5	2	1024	896	128	768	88	128	true
mod-lists:1.0.0	5	2	3000	2600	128	2048	384	512	false

Methodology

Enable R/W split for mod-fqm-manager.
Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent users.
Prepare 200K item records for the query to return.
Conduct tests with JMeter script (single tenant).

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

[Poppy] List App with single tenant and R/W split enabled