1 Overview
2 Summary
3 Recommendations and Jiras
4 Results
- 4.1 Memory Utilization
- 4.2 Service CPU Utilization
- 4.3 RDS CPU Utilization
- 4.4 RDS Database Connections
- 4.5 DB Load
5 Appendix
- 5.1 Infrastructure
- 5.2 Methodology/Approach

Overview

This document contains the results of testing List App refreshing of 200k records in the Orchid release. https://folio-org.atlassian.net/browse/PERF-619

Summary

Duration for list refresh of 200k records is up to 11 minutes 40 seconds for 10 concurrent users.
Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.
For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules.
Approximately DB CPU usage is up to 99.5%.

Recommendations and Jiras

Use database read-write split enabled for mod-lists to improve performance.
Increase default CPU allocation for mod-list service up to 4x to 512 from 128.

Results

200k records list refresh for each user.

Number of users	Duration Test 1	Duration Test 2	Duration Test 3

Number of users	Duration Test 1	Duration Test 2	Duration Test 3
mod-lists R/W split enabled
1 user	13 min 30 sec (1st time after mod-lists restart)	4min 33 sec	4 min 52 sec
2 users	4 min 40 sec	5 min	4 min 55 sec
5 users	6 min-6 min 13 sec	6 min 16 sec-6 min 24 sec	6min 41 sec
10 users	9 min 40 sec - 10 min 10 sec	10 min -10 min 40 sec	9 min 50 sec-10 min 7 sec
mod-lists R/W split disabled
1 user	13 min 23 sec (1st time after mod-lists restart)	4 min 30 sec	4 min 24 sec
2 users	4 min 42 sec	5 min 15 sec	5 min 19 sec
5 users	7 min -7 min 29 sec	6 min 55 sec- 7 min 24 sec	7 min -7 min 25 sec
10 users	10 min 30 sec - 11 min 35 sec	11 min 10 sec- 11 min 40 sec	10 min 45 sec- 11 min 35 sec
mod-lists R/W split disabled + increased CPU to 512 for mod-lists (CPU usage decreased to 84%)
1 user	5 min 3 sec (1st time after mod-lists restart)	4 min 55 sec	5 min
2 users	5 min	5 min 42 sec	5 min 25 sec
5 users	7 min 30 sec- 8 min	7 min 10 sec- 7 min 30 sec	7 min 10 sec- 7 min 30 sec
10 users	10 min -10 min 45 sec	10 min 10 sec -11 min	10 min 45 sec- 10 min 55 sec

* - Tests were performed with disabled circulation jobs

Memory Utilization

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.

mod-lists

Service CPU Utilization

For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.

mod-lists

RDS CPU Utilization

RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%

RDS Database Connections

DB Load

Long-running queries:

select id from [tenant]_mod_fqm_manager.drv_item_details where lower(cast(item_status as varchar)) = lower($1) order by item_effective_call_number asc, item_effective_location_name asc, instance_title asc, instance_primary_contributor asc

delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

Records count :

mod_inventory_storage.item = 22130108

PTF -environment ncp5

12 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module ncp5-pvt Wed Aug 23 10:38:27 UTC 2023	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled

Module ncp5-pvt Wed Aug 23 10:38:27 UTC 2023	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage	14	mod-remote-storage:2.0.3	2	4920	4472	1024	3960	512	512	true
mod-codex-mux	3	mod-codex-mux:2.13.0-SNAPSHOT.108	2	1024	896	128	768	88	128	false
mod-finance-storage	9	mod-finance-storage:8.4.2	2	1024	896	1024	700	88	128	false
mod-ncip	9	mod-ncip:1.13.1	2	1024	896	128	768	88	128	false
mod-agreements	6	mod-agreements:5.5.2	2	1592	1488	128	968	384	512	false
mod-ebsconet	6	mod-ebsconet:2.0.0	2	1248	1024	128	700	128	256	false
mod-organizations	6	mod-organizations:1.7.0	2	1024	896	128	700	88	128	false
edge-sip2	4	edge-sip2:3.0.0	2	1024	896	128	768	88	128	false
mod-settings	6	mod-settings:1.0.0	2	1024	896	200	0	0	0	false
mod-data-import	9	mod-data-import:2.7.1	1	2048	1844	256	1292	384	512	true
edge-dematic	14	edge-dematic:2.0.1	1	1024	896	128	0	0	0	false
mod-search	8	mod-search:2.0.1	2	2592	2480	2048	1440	512	1024	false
mod-inn-reach	1	mod-inn-reach:3.1.0-SNAPSHOT.49	2	2304	2048	512	1440	512	1024	false
mod-tags	6	mod-tags:2.0.1	2	1024	896	128	768	88	128	false
mod-authtoken	7	mod-authtoken:2.13.0	2	1440	1152	512	922	88	128	true
mod-notify	7	mod-notify:3.0.0	2	1024	896	128	768	88	128	false
mod-inventory-update	6	mod-inventory-update:3.0.1	2	1024	896	128	768	88	128	false
mod-configuration	9	mod-configuration:5.9.1	2	1024	896	128	768	88	128	true
mod-orders-storage	6	mod-orders-storage:13.5.0	2	1024	896

[Orchid] List App test report