Overview

This document contains the results of testing List App refreshing of 200k records in the Orchid release. PERF-619 - Getting issue details... STATUS

Summary

Duration for list refresh of 200k records is up to 11 minutes 40 seconds for 10 concurrent users.
Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.
For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules.
Approximately DB CPU usage is up to 99.5%.

Recommendations and Jiras

Use database read-write split enabled for mod-lists to improve performance.
Increase default CPU allocation for mod-list service up to 4x to 512 from 128.

Results

200k records list refresh for each user.

Number of users	Duration Test 1	Duration Test 2	Duration Test 3
mod-lists R/W split enabled
1 user	13 min 30 sec (1st time after mod-lists restart)	4min 33 sec	4 min 52 sec
2 users	4 min 40 sec	5 min	4 min 55 sec
5 users	6 min-6 min 13 sec	6 min 16 sec-6 min 24 sec	6min 41 sec
10 users	9 min 40 sec - 10 min 10 sec	10 min -10 min 40 sec	9 min 50 sec-10 min 7 sec
mod-lists R/W split disabled
1 user	13 min 23 sec (1st time after mod-lists restart)	4 min 30 sec	4 min 24 sec
2 users	4 min 42 sec	5 min 15 sec	5 min 19 sec
5 users	7 min -7 min 29 sec	6 min 55 sec- 7 min 24 sec	7 min -7 min 25 sec
10 users	10 min 30 sec - 11 min 35 sec	11 min 10 sec- 11 min 40 sec	10 min 45 sec- 11 min 35 sec
mod-lists R/W split disabled + increased CPU to 512 for mod-lists
1 user	5 min 3 sec (1st time after mod-lists restart)	4 min 55 sec	5 min
2 users	5 min	5 min 42 sec	5 min 25 sec
5 users	7 min 30 sec- 8 min	7 min 10 sec- 7 min 30 sec	7 min 10 sec- 7 min 30 sec
10 users	10 min -10 min 45 sec	10 min 10 sec -11 min	10 min 45 sec- 10 min 55 sec

* - Tests were performed with disabled circulation jobs

Memory Utilization

Memory utilization increasing for mod-lists up to 33% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all of the modules.

mod-lists

Service CPU Utilization

For mod-lists, CPU utilization was up to 384% for the test with 10 concurrent users and R/W split disabled. Average CPU usage did not exceed 13 % for all other modules. Each CPU spike corresponds to a separate list refresh.

mod-lists

RDS CPU Utilization

RDS CPU Utilization was very high. Approximately DB CPU usage is up to 99.5%

RDS Database Connections

DB Load

Long-running queries:

select id from [tenant]_mod_fqm_manager.drv_item_details where lower(cast(item_status as varchar)) = lower($1) order by item_effective_call_number asc, item_effective_location_name asc, instance_title asc, instance_primary_contributor asc

delete from list_contents where list_id=$1 and refresh_id=$2

Appendix

Infrastructure

Records count :

mod_inventory_storage.item = 22130108

PTF -environment ncp5

12 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module ncp5-pvt Wed Aug 23 10:38:27 UTC 2023	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage	14	mod-remote-storage:2.0.3	2	4920	4472	1024	3960	512	512	true
mod-codex-mux	3	mod-codex-mux:2.13.0-SNAPSHOT.108	2	1024	896	128	768	88	128	false
mod-finance-storage	9	mod-finance-storage:8.4.2	2	1024	896	1024	700	88	128	false
mod-ncip	9	mod-ncip:1.13.1	2	1024	896	128	768	88	128	false
mod-agreements	6	mod-agreements:5.5.2	2	1592	1488	128	968	384	512	false
mod-ebsconet	6	mod-ebsconet:2.0.0	2	1248	1024	128	700	128	256	false
mod-organizations	6	mod-organizations:1.7.0	2	1024	896	128	700	88	128	false
edge-sip2	4	edge-sip2:3.0.0	2	1024	896	128	768	88	128	false
mod-settings	6	mod-settings:1.0.0	2	1024	896	200	0	0	0	false
mod-data-import	9	mod-data-import:2.7.1	1	2048	1844	256	1292	384	512	true
edge-dematic	14	edge-dematic:2.0.1	1	1024	896	128	0	0	0	false
mod-search	8	mod-search:2.0.1	2	2592	2480	2048	1440	512	1024	false
mod-inn-reach	1	mod-inn-reach:3.1.0-SNAPSHOT.49	2	2304	2048	512	1440	512	1024	false
mod-tags	6	mod-tags:2.0.1	2	1024	896	128	768	88	128	false
mod-authtoken	7	mod-authtoken:2.13.0	2	1440	1152	512	922	88	128	true
mod-notify	7	mod-notify:3.0.0	2	1024	896	128	768	88	128	false
mod-inventory-update	6	mod-inventory-update:3.0.1	2	1024	896	128	768	88	128	false
mod-configuration	9	mod-configuration:5.9.1	2	1024	896	128	768	88	128	true
mod-orders-storage	6	mod-orders-storage:13.5.0	2	1024	896	512	700	88	128	false
edge-caiasoft	6	edge-caiasoft:2.0.0	2	1024	896	128	0	0	0	false
mod-login-saml	6	mod-login-saml:2.6.2	2	1024	896	128	768	88	128	false
mod-erm-usage-harvester	7	mod-erm-usage-harvester:4.3.1	2	1024	896	128	768	88	128	false
mod-password-validator	6	mod-password-validator:3.0.0	2	1440	1298	128	768	384	512	false
mod-licenses	6	mod-licenses:4.3.1	2	2480	2312	128	1792	384	512	false
mod-gobi	6	mod-gobi:2.6.0	2	1024	896	128	700	88	128	false
mod-bulk-operations	17	mod-bulk-operations:1.0.6	2	3072	2600	1024	1536	384	512	false
mod-fqm-manager	19	mod-fqm-manager:1.0.0-SNAPSHOT.841	1	3000	2600	128	2048	384	512	true
mod-graphql	6	mod-graphql:1.11.0	2	1024	896	128	768	88	128	false
mod-finance	6	mod-finance:4.7.1	2	1024	896	128	700	88	128	false
mod-erm-usage	7	mod-erm-usage:4.5.3	2	1024	896	128	768	88	128	false
mod-copycat	6	mod-copycat:1.4.0	2	1024	896	128	768	88	128	false
mod-lists	3-4-5	mod-lists:1.0.0-SNAPSHOT.1130	1	3000	2600	128/512	2048	384	512	false/true
mod-permissions	20	mod-permissions:6.3.2	2	1684	1544	512	1024	384	512	true
mod-entities-links	7	mod-entities-links:1.0.2	2	2592	2480	400	1440	0	1024	false
pub-edge	7	pub-edge:2023.06.14	2	1024	896	128	768	0	0	false
mod-codex-inventory	3	mod-codex-inventory:2.4.0-SNAPSHOT.139	2	1024	896	128	768	88	128	true
mod-orders	11	mod-orders:12.6.8	2	2048	1440	1024	1024	384	512	false
edge-patron	6	edge-patron:4.11.0	2	1024	896	256	768	88	128	false
edge-ncip	6	edge-ncip:1.8.1	2	1024	896	128	768	88	128	false
mod-users-bl	8	mod-users-bl:7.5.0	2	1440	1152	512	922	88	128	false
mod-inventory-storage	11	mod-inventory-storage:26.0.0	2	4096	3690	2048	3076	384	512	true
mod-invoice	6	mod-invoice:5.6.5	2	1440	1152	512	922	88	128	false
mod-user-import	6	mod-user-import:3.7.2	2	1024	896	128	768	88	128	false
mod-sender	7	mod-sender:1.10.0	2	1024	896	128	768	88	128	false
edge-oai-pmh	9	edge-oai-pmh:2.6.1	2	1512	1360	1024	1440	384	512	false
mod-data-export-worker	10	mod-data-export-worker:3.0.13	2	3072	2800	1024	2048	384	512	false
mod-rtac	8	mod-rtac:3.5.0	2	1024	896	128	768	88	128	false
mod-task-list	3	mod-task-list:1.7.1	2	1024	896	128	768	88	128	false
mod-circulation-storage	13	mod-circulation-storage:16.0.0.16	2	2880	2592	1536	1814	384	512	false
mod-calendar	8	mod-calendar:2.4.2	2	1024	896	128	768	88	128	false
mod-source-record-storage	26	mod-source-record-storage:5.6.7	2	5600	5000	2048	3500	384	512	true
mod-event-config	6	mod-event-config:2.5.0	2	1024	896	128	768	88	128	false
mod-courses	6	mod-courses:1.4.7	2	1024	896	128	768	88	128	false
mod-inventory	1	mod-inventory:20.0.4	0	2880	2592	1024	1814	384	512	false
mod-email	6	mod-email:1.15.3	2	1024	896	128	768	88	128	false
mod-circulation	7	mod-circulation:23.5.6	2	2880	2592	1536	768	88	128	false
mod-di-converter-storage	6	mod-di-converter-storage:2.0.5	2	1024	896	128	768	88	128	false
mod-pubsub	10	mod-pubsub:2.9.1	2	1536	1440	1024	922	384	512	true
edge-orders	6	edge-orders:2.8.1	2	1024	896	128	768	88	128	false
edge-rtac	6	edge-rtac:2.6.0	2	1024	896	128	768	88	128	false
mod-template-engine	6	mod-template-engine:1.18.0	2	1024	896	128	768	88	128	false
mod-users	8	mod-users:19.1.1	2	1024	896	128	768	88	128	true
mod-patron-blocks	11	mod-patron-blocks:1.8.0	2	1024	896	1024	768	88	128	true
mod-audit	9	mod-audit:2.7.0	2	1024	896	128	768	88	128	false
mod-source-record-manager	18	mod-source-record-manager:3.6.4	2	5600	5000	2048	3500	384	512	false
nginx-edge	8	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	false
mod-quick-marc	6	mod-quick-marc:3.0.0	1	2288	2176	128	1664	384	512	false
nginx-okapi	7	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	false
okapi-b	8	okapi:5.0.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	9	mod-feesfines:18.2.1	2	1024	896	128	768	88	128	true
mod-invoice-storage	6	mod-invoice-storage:5.6.0	2	1872	1536	1024	1024	384	512	false
mod-codex-ekb	3	mod-codex-ekb:1.11.0-SNAPSHOT.141	2	1024	896	128	768	88	128	false
mod-service-interaction	6	mod-service-interaction:2.2.2	2	2048	1844	256	1290	384	512	false
mod-data-export	6	mod-data-export:4.7.1	1	1024	896	1024	768	88	128	false
mod-patron	7	mod-patron:5.5.2	2	1024	896	128	768	88	128	false
mod-oai-pmh	10	mod-oai-pmh:3.11.3	2	4096	3690	2048	3076	384	512	true
edge-connexion	6	edge-connexion:1.0.6	2	1024	896	128	768	88	128	false
mod-kb-ebsco-java	7	mod-kb-ebsco-java:3.13.0	2	1024	896	128	768	88	128	false
mod-notes	6	mod-notes:5.0.1	2	1024	896	128	952	384	512	false
mod-data-export-spring	8	mod-data-export-spring:2.0.2	1	2048	1844	256	1536	384	512	false
mod-organizations-storage	7	mod-organizations-storage:4.5.1	2	1024	896	128	700	88	128	false
mod-login	6	mod-login:7.9.0	2	1440	1298	1024	768	384	512	false
pub-okapi	7	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	false
mod-data-import-converter-storage	4	mod-data-import-converter-storage:1.16.0-SNAPSHOT.132	2	1024	896	128	768	88	128	false
mod-eusage-reports	7	mod-eusage-reports:1.3.0	2	1024	896	128	768	88	128	false

Methodology/Approach

To test Baseline for List refresh JMeter scripts were used.

Test preparation:

Background circulation jobs were disabled.
200k items were checked out
10 lists were created with the query: (item_status == "Checked out") to be able to run a test for 10 concurrent users.

Refresh list for 1, 2, 5, and 10 concurrent users tested

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

PTF - List App (Orchid)