1 Overview
2 Summary
3 Test runs
4 Results
- 4.1 CPU Utilization
  - 4.1.1 LA_test_1
  - 4.1.2 LA_test_5
  - 4.1.3 LA_test_10
- 4.2 Memory Utilization
  - 4.2.1 LA_test_1.
  - 4.2.2 LA_test_5
  - 4.2.3 LA_test_10
- 4.3 DB CPU Utilization
- 4.4 DB Connections
- 4.5 DB Load
5 Appendix
- 5.1 Infrastructure
- 5.2 Methodology
- 5.3 Additional Screenshots of graphs or charts (Optional)

Overview

This document contains the results of testing Lists App refreshing multiple concurrent lists with different queries MODLISTS-58 on a single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of the Lists App modules (mod-lists, mod-fqm-manager) with load from across multiple concurrent users.

Ticket: https://folio-org.atlassian.net/browse/PERF-811

Summary

All tests were finished successfully. List for query (item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_location_name not in [acq admin, Administration Office]) AND (item_status == Available) AND (item_material_type == book) AND (item_updated_date > 2019-07-01) was not created because of long creation duration(> 5 hours)
Performance testing of Lists App refreshing was performed for 1, 5, and 10 parallel threads (virtual users). With the increase in the number of threads, the duration increased in some cases by 4 times.
For all 3 tests, the services CPU utilization has about the same value. It reached a maximum of 40% for mod-fqm-manager and 35% for mod-lists. No memory leak is suspected for all the modules.
Database CPU utilization on LA_test_5 and LA_test_10 was a maximum of 95% on the reader instance and 60% on the writer instance. Reader instance is a bottleneck

Test runs

LA_test_1 - was executed to collect baseline metrics. 30 lists were updated one after another, from first to last. Number of loops =3;

LA_test_5 - load test for 5 concurrent threads;

LA_test_10 - load test for 10 concurrent threads.

Results

	List-APP Query	List ID	Record count	Avarage duration, seconds LA_test_1	Duration, sec 5VU LA_test_5	Duration, sec 10VU LA_test_10

	List-APP Query	List ID	Record count	Avarage duration, seconds LA_test_1	Duration, sec 5VU LA_test_5	Duration, sec 10VU LA_test_10
1	item_status in [Aged to lost, Claimed returned, Declared lost, Long missing, Missing]	fb9f3393-4383-44d2-b9e7-1680aa42c6fd	53870	45	89	137
2	item_status != Available	49da7c31-936f-4044-ae0b-8defb049f2e7	194923	228	259	294
3	instance_updated_date > 2019-07-20	369cbfd4-e48f-4e0d-8a41-aad011d0d705	815097	2046	3521	2189
4	instance_primary_contributor contains Smith	4152d2df-9467-46d9-a4a2-6e063cf63c1b	26453	3279	2902	4500
5	item_permanent_location_name == acq_admin(item_permanent_location_name == acq admin) AND (item_status in [Available, Awaiting delivery, Awaiting pickup])	b8e3797b-b8c8-4269-8fc9-96cae17804be	9999	42	83	152
6	item_material_type not In book, electronic resource	9ddc4e10-3466-457d-8ddb-d7c5a44dfb1a	8285	152	222	262
7	item_copy_number != c.1	25e4949f-cd73-4785-bcc3-0d4f58810b93	539634	585	932	834
8	(item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_call_number starts with D)	d5ffead3-969c-4db4-83c5-361919580e62	851761	1008	1793	2940
9	(item_material_type == book) AND (item_created_date >= 2018-01-01) AND (item_created_date <= 2021-01-01)	063beade-a5e7-4127-bea4-6104c2956cda	119057	114	215	174
10	instance_id in [list]	87762249-5c9b-4165-aafd-2707e1e65388	13	957	1185	1056
11	item_permanent_location_name == acq admin	83856618-c2fb-4fe7-b81a-fe03451c6e90	10016	41	42	132
12	(instance_id not in [970a8b97-d58c-46e4-9766-f777a99abc2c, 7ef91348-858a-4860-925f-77077410249d, 9796d69d-0acf-429d-a23f-eaa5aac84c7c]) AND (item_status not in [Available, In transit, Paged, Restricted])	2efb4077-83d4-4dcb-bda8-d5d869153739	194923	213	212	457
13	instance_title contains africa	55b1aa25-245a-409e-9849-a4ef1c6034c1	65085	3495	2934	canceled
14	id in [list[100]]	0c37b4d0-93f1-4cfd-a359-e97d239b7e8a	100	880	827	1102
15	(item_permanent_location_name == UC/HP/ASR/JRLASR) AND (item_temporary_location_name != UC/HP/ASR/JRLASR)	06c87637-00a6-43ed-b9ff-9c6bc243cd01	14965	43	43	123
16	item_status == Checked out	fd201d94-6ae5-46bd-836f-a1836bc2c79a	55449	159	226	34
17	(user_active == false) AND (loan_status == Open)	0885f86c-1ffc-4264-9b15-d6555228ad53	213	10	10	10
18	(loan_status == Open) AND (item_status == Available)	30a8b2ea-6510-4a41-9602-1f1e05e91b54	0	30	40	200
19	(loan_checkout_date > 2020-01-01) AND (loan_checkout_date < 2021-01-01)	accb9b80-b859-4fa5-abc6-38bae090f18c	5158	11	11	11
20	loan_policy_name in [stksq3r, stksqunlim, stksyearly]	845dd6a0-e0a4-43e3-bb6a-087b415da2ce	31183	17	21	18
21	(loan_status == Open) AND (user_expiration_date <= 2023-12-31)	a9b70301-fb06-4514-9bc1-feb76548864e	4109	10	11	41
22	loan_status == Open	99d1b3c3-98df-4221-b8e8-1c7ae74796e1	55449	25	115	80
23	(loan_status == Open) AND (user_patron_group not in [Faculty, Staff]) AND (item_material_type == book)	8d0a6db9-316c-4103-81ed-6f969b622932	55449	24	120	49
24	user_active == true	2b69b854-b440-4f6f-b1d8-a3bb606f10ec	84899	50	42	42
25	user_patron_group not in [Undergraduate, Graduate]	01378c7c-ae29-4944-90da-646def8140e0	90625	41	51	67
26	user_patron_group in [Undergraduate, Graduate]	48943943-5ce2-44a7-9bed-b2af4d12f1f4	17	10	10	10
27	user_primary_address contains CA	89dccdbe-4a0c-4c26-b3f9-88924b7d5be4	15886	13	15	14
28	user_preferred_contact_type not in [Email, Text Message]	e2e9dd7a-d069-4ade-85f7-a0b61696bf98	90016	45	57	38
29	user_barcode starts with 0	bb641ed1-49d6-4a35-ae99-b92add32ed0f	90040	38	68	44

CPU Utilization

LA_test_1

During the LA_test_1 test CPU utilization reached a maximum of 33% for mod-fqm-manager and 30% for mod-lists.

LA_test_5

During the test CPU utilization reached a maximum of 40% for mod-fqm-manager and 30% for mod-lists.

LA_test_10

During the test CPU utilization reached a maximum of 40% for mod-fqm-manager and 37% for mod-lists.

Memory Utilization

No memory leak is suspected for all the modules during all tests.

LA_test_1.

The mod-fqm-manager service increased the use of CPU from 64% to 84% and mod-lists from 62% to 68%

LA_test_5

The mod-fqm-manager service increased the use of CPU from 86% to 92% and mod-lists from 74% to 76%

LA_test_10

The mod-fqm-manager service increased the use of CPU from 87% to 92% and mod-lists from 76% to 77%

DB CPU Utilization

Maximum DB CPU utilization reached 57% (writer instance) and 82% (reader instance) during the LA_test_1 test.

Maximum DB CPU utilization reached 60% (writer instance) and 95% (reader instance) during the LA_test_5 test.

Maximum DB CPU utilization reached 73% (writer instance) and 94% (reader instance) during the LA_test_10 test.

DB Connections

During all tests, the number of connections to the reader instance = 20; the number of connections to the writer instance ~400;

LA_test_1

LA_test_5

LA_test_10

DB Load

LA_test_1

Writer instance

Reader instance

LA_test_5

Writer instance

Reader instance

LA_test_10

Writer instance

Reader instance

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
DB Cluster with 2 instances (reader and writer)
NameAPI NameMemory GIBvCPUsmax_connectionsR6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
MSK tenant
4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcp1-pvt

mod-lists	2	mod-lists:1.1.0-SNAPSHOT.1266	2	1024	896	128	768	88	128	FALSE
mod-fqm-manager	2	mod-fqm-manager:1.1.0-SNAPSHOT.1083	2	1024	896	128	768	88	128	FALSE
mod-inventory-storage	2	mod-inventory-storage:27.1.0-SNAPSHOT.719	2	4096	3690	2048	3076	384	512	FALSE
mod-circulation-storage	2	mod-circulation-storage:17.2.0-SNAPSHOT.384	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	2	mod-source-record-storage:5.8.0-SNAPSHOT.353	2	5600	5000	2048	3500	384	512	FALSE
mod-circulation	2	mod-circulation:24.2.0-SNAPSHOT.649	2	2880	2592	1536	1814	384	512	FALSE
mod-users	2	mod-users:19.3.0-SNAPSHOT.679	2	1024	896	128	768	88	128	FALSE
okapi-b	2	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE

Methodology

Enable R/W split for mod-fqm-manager and mod-lists and add reader instance to DB cluster

{
                    "name": "DB_HOST_READER",
                    "value": "db-ro.folio-eis.us-east-1"
                },
                 {
                    "name": "DB_PORT_READER",
                    "value": "5432"
                },

All of the lists were created manually and data were generated to meet the requirements for the number of returned records
Conduct tests with JMeter script (main tenant).
Duration of refresh lists were extracted from DB using SQL query
SELECT id, list_id, status, refresh_start_date, refresh_end_date,(refresh_end_date - refresh_start_date)as duration, records_count FROM fs09000000_mod_lists.list_refresh_details where refresh_end_date <='"2024-03-26 20:18:30"' and refresh_start_date >='"2024-03-26 16:57:00"' Order by refresh_start_date asc limit 29

Additional Screenshots of graphs or charts (Optional)

Table with the number of returned records count by each of the list and requested

№	Entity	Query	User friendly query	Approx record count	Operators used	DB record count
1	item	{"item_status":{"$in":["Aged to lost","Claimed returned","Declared lost","Long missing","Missing"]}}	item_status in [Aged to lost, Claimed returned, Declared lost, Long missing, Missing]	52k	in	53870
2	item	{"item_status":{"$ne":"Available"}}	item_status != Available	137k	ne	192676
3	item	{"instance_updated_date":{"$gt":"2019-07-20"}}	instance_updated_date > 2019-07-20	848k	gt	815097
4	item	{"instance_primary_contributor":{"$regex":"Smith"}}	instance_primary_contributor contains Smith	18k	regex contains	26453
5	item	{"$and":[{"item_permanent_location_id":{"$eq":"f4619e23-d081-4447-a589-e278037e7f5e"}},{"item_status":{"$in":["Available","Awaiting delivery","Awaiting pickup"]}}]}	item_permanent_location_name == acq_admin(item_permanent_location_name == acq admin) AND (item_status in [Available, Awaiting delivery, Awaiting pickup])	10k	and, eq, in	9999
6	item	{"item_material_type_id":{"$nin":["025ba2c5-5e96-4667-a677-8186463aee69","1aa8050a-619b-40c6-9b9b-81a2aa9c0912"]}}	item_material_type not In book, electronic resource	6k	nin	60069
7	item	{"item_copy_number":{"$ne":"c.1"}}	item_copy_number != c.1	485k	ne	539634
8	item	{"$and":[{"item_effective_call_number_typeid":{"$eq":"c73c17f4-9660-421b-bdfc-e1c6093dec13"}},{"item_effective_location_id":{"$nin":["f4619e23-d081-4447-a589-e278037e7f5e","b1dbe1fa-d76e-4db0-96b4-62c4a0674b29"]}},{"item_status":{"$eq":"Available"}},{"item_material_type_id":{"$eq":"025ba2c5-5e96-4667-a677-8186463aee69"}},{"item_updated_date":{"$gt":"2019-07-01"}}]}	(item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_location_name not in [acq admin, Administration Office]) AND (item_status == Available) AND (item_material_type == book) AND (item_updated_date > 2019-07-01)	428k	and, eq, nin, gt	NOT Created
9	item	{"$and":[{"item_effective_call_number_typeid":{"$eq":"c73c17f4-9660-421b-bdfc-e1c6093dec13"}},{"item_effective_call_number":{"$regex":"^D"}}]}	(item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_call_number starts with D)	817k	and, eq, regex starts with	851761
10	item	{"$and":[{"item_material_type_id":{"$eq":"025ba2c5-5e96-4667-a677-8186463aee69"}},{"item_created_date":{"$gte":"2018-01-01"}},{"item_created_date":{"$lte":"2021-01-01"}}]}	(item_material_type == book) AND (item_created_date >= 2018-01-01) AND (item_created_date <= 2021-01-01)	202k	and, eq, gte, lte	119056
11	item	{"instance_id":{"$in":["“]	instance_id in []	21	in	13
12	item	{"item_permanent_location_id":{"$eq":"f4619e23-d081-4447-a589-e278037e7f5e"}}	item_permanent_location_name == acq admin	10k	eq	10016
13	item	{"$and":[{"instance_id":{"$nin":["970a8b97-d58c-46e4-9766-f777a99abc2c","7ef91348-858a-4860-925f-77077410249d","9796d69d-0acf-429d-a23f-eaa5aac84c7c"]}},{"item_status":{"$nin":["Available","In transit","Paged","Restricted"]}}]}	(instance_id not in [970a8b97-d58c-46e4-9766-f777a99abc2c, 7ef91348-858a-4860-925f-77077410249d, 9796d69d-0acf-429d-a23f-eaa5aac84c7c]) AND (item_status not in [Available, In transit, Paged, Restricted])	102k	and, nin	192676
14	item	{"instance_title":{"$regex":"africa"}}	instance_title contains africa	38k	regex contains	65085
15	item	{"id":{"$in":[""]}}	id in []	100	in	100
16	item	{"$and":[{"item_permanent_location_id":{"$eq":"c3dd9997-463b-47e3-958c-2c6fc2775f90"}},{"item_temporary_location_id":{"$ne":"c3dd9997-463b-47e3-958c-2c6fc2775f90"}}]}	(item_permanent_location_name == UC/HP/ASR/JRLASR) AND (item_temporary_location_name != UC/HP/ASR/JRLASR)	3	eq, neq	14965
17	item	{"item_status":{"$eq":"Checked out"}}	item_status == Checked out	4k	eq	55449
18	loan	{"$and":[{"user_active":{"$eq":"false"}},{"loan_status":{"$eq":"Open"}}]}	(user_active == false) AND (loan_status == Open)	1k	and, eq	213
19	loan	{"$and":[{"loan_status":{"$eq":"Open"}},{"item_status":{"$eq":"Available"}}]}	(loan_status == Open) AND (item_status == Available)	0	and, eq	0
20	loan	{"$and":[{"loan_checkout_date":{"$gt":"2020-01-01"}},{"loan_checkout_date":{"$lt":"2021-01-01"}}]}	(loan_checkout_date > 2020-01-01) AND (loan_checkout_date < 2021-01-01)	20k	and, gt, lt	5158
21	loan	{"loan_policy_id":{"$in":["301ccbe5-fc1a-4a6e-adcb-ed55fc8dc9cf","26a6fc3d-c437-40ec-8cbe-ea9e19322319","b959df2b-c9ce-4b78-8e7f-38bde929c681"]}}	loan_policy_name in [stksq3r, stksqunlim, stksyearly]	33k	in	31183
22	loan	{"$and":[{"loan_status":{"$eq":"Open"}},{"user_expiration_date":{"$lt":"2023-12-31"}}]}	(loan_status == Open) AND (user_expiration_date <= 2023-12-31)	1k	and, eq, lte	4109
23	loan	{"loan_status":{"$eq":"Open"}}	loan_status == Open	50k	eq	55449
24	loan	{"$and":[{"loan_status":{"$eq":"Open"}},{"user_patron_group_id":{"$nin":["d01b7bba-48fb-4264-833b-b88d23505404","d010bfa2-4983-4741-9ef8-1295484297cc"]}},{"item_material_type_id":{"$eq":"025ba2c5-5e96-4667-a677-8186463aee69"}}]}'	(loan_status == Open) AND (user_patron_group not in [Faculty, Staff]) AND (item_material_type == book)	50k	and, eq, nin	55449
25	user	{"user_active":{"$eq":"true"}}	user_active == true	90k	eq	84899
26	user	{"user_patron_group_id":{"$nin":["6ad290aa-5851-4356-bb8d-1015e40c05c2","ba484dd9-a2c3-4d80-b067-91cd2066134b"]}}	user_patron_group not in [Undergraduate, Graduate]	90k	nin	90625
27	user	user_patron_group in [Undergraduate, Graduate]	user_patron_group in [Undergraduate, Graduate]	100	in	17
28	user	{"user_primary_address":{"$regex":"CA"}}	user_primary_address contains CA	16k	regex contains	15886
29	user	{"user_preferred_contact_type":{"$nin":["Email","Text Message"]}}	user_preferred_contact_type not in [Email, Text Message]	90k	nin	90016
30	user	{"user_barcode":{"$regex":"^0"}}	user_barcode starts with 0	90k	regex starts with	90016

The first test LA_test_1 - was executed to collect baseline metrics. 30 lists were updated one after another, from first to last. Number of loops =3, detailed information about each run is in the table

Query	Query	Refresh list status	Record count	Loop 1. Duration, seconds	Loop 2. Duration, seconds	Loop 3. Duration, seconds	Avarage duration, seconds
item_status in [Aged to lost, Claimed returned, Declared lost, Long missing, Missing]	fb9f3393-4383-44d2-b9e7-1680aa42c6fd	SUCCESS	53870	25	54	56	45
item_status != Available	49da7c31-936f-4044-ae0b-8defb049f2e7	SUCCESS	194923	276	162	246	228
instance_updated_date > 2019-07-20	369cbfd4-e48f-4e0d-8a41-aad011d0d705	SUCCESS	815097	2006	2020	2112	2046
instance_primary_contributor contains Smith	4152d2df-9467-46d9-a4a2-6e063cf63c1b	SUCCESS	26453	3274	3286	3276	3279
item_permanent_location_name == acq_admin(item_permanent_location_name == acq admin) AND (item_status in [Available, Awaiting delivery, Awaiting pickup])	b8e3797b-b8c8-4269-8fc9-96cae17804be	SUCCESS	9999	42	42	41	42
item_material_type not In book, electronic resource	9ddc4e10-3466-457d-8ddb-d7c5a44dfb1a	SUCCESS	8285	152	152	153	152
item_copy_number != c.1	25e4949f-cd73-4785-bcc3-0d4f58810b93	SUCCESS	539634	496	482	777	585
(item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_call_number starts with D)	d5ffead3-969c-4db4-83c5-361919580e62	SUCCESS	851761	1074	1061	890	1008
(item_material_type == book) AND (item_created_date >= 2018-01-01) AND (item_created_date <= 2021-01-01)	063beade-a5e7-4127-bea4-6104c2956cda	SUCCESS	119057	113	120	108	114
instance_id in [list]	87762249-5c9b-4165-aafd-2707e1e65388	SUCCESS	13	898	987	986	957
item_permanent_location_name == acq admin	83856618-c2fb-4fe7-b81a-fe03451c6e90	SUCCESS	10016	42	41	41	41
(instance_id not in [970a8b97-d58c-46e4-9766-f777a99abc2c, 7ef91348-858a-4860-925f-77077410249d, 9796d69d-0acf-429d-a23f-eaa5aac84c7c]) AND (item_status not in [Available, In transit, Paged, Restricted])	2efb4077-83d4-4dcb-bda8-d5d869153739	SUCCESS	194923	272	177	189	213
instance_title contains africa	55b1aa25-245a-409e-9849-a4ef1c6034c1	SUCCESS	65085	3488	3500	3498	3495
id in [list[100]]	0c37b4d0-93f1-4cfd-a359-e97d239b7e8a	SUCCESS	100	827	905	908	880
(item_permanent_location_name == UC/HP/ASR/JRLASR) AND (item_temporary_location_name != UC/HP/ASR/JRLASR)	06c87637-00a6-43ed-b9ff-9c6bc243cd01	SUCCESS	14965	44	43	42	43
item_status == Checked out	fd201d94-6ae5-46bd-836f-a1836bc2c79a	SUCCESS	55449	164	153	160	159
(user_active == false) AND (loan_status == Open)	0885f86c-1ffc-4264-9b15-d6555228ad53	SUCCESS	213	10	10	10	10
(loan_status == Open) AND (item_status == Available)	30a8b2ea-6510-4a41-9602-1f1e05e91b54	SUCCESS	0	30	31	30	30
(loan_checkout_date > 2020-01-01) AND (loan_checkout_date < 2021-01-01)	accb9b80-b859-4fa5-abc6-38bae090f18c	SUCCESS	5158	12	10	10	11
loan_policy_name in [stksq3r, stksqunlim, stksyearly]	845dd6a0-e0a4-43e3-bb6a-087b415da2ce	SUCCESS	31183	19	17	14	17
(loan_status == Open) AND (user_expiration_date <= 2023-12-31)	a9b70301-fb06-4514-9bc1-feb76548864e	SUCCESS	4109	10	10	10	10
loan_status == Open	99d1b3c3-98df-4221-b8e8-1c7ae74796e1	SUCCESS	55449	29	26	21	25
(loan_status == Open) AND (user_patron_group not in [Faculty, Staff]) AND (item_material_type == book)	8d0a6db9-316c-4103-81ed-6f969b622932	SUCCESS	55449	19	27	25	24
user_active == true	2b69b854-b440-4f6f-b1d8-a3bb606f10ec	SUCCESS	84899	58	46	46	50
user_patron_group not in [Undergraduate, Graduate]	01378c7c-ae29-4944-90da-646def8140e0	SUCCESS	90625	55	33	34	41
user_patron_group in [Undergraduate, Graduate]	48943943-5ce2-44a7-9bed-b2af4d12f1f4	SUCCESS	17	10	10	10	10
user_primary_address contains CA	89dccdbe-4a0c-4c26-b3f9-88924b7d5be4	SUCCESS	15886	13	13	13	13
user_preferred_contact_type not in [Email, Text Message]	e2e9dd7a-d069-4ade-85f7-a0b61696bf98	SUCCESS	90016	52	38	45	45
user_barcode starts with 0	bb641ed1-49d6-4a35-ae99-b92add32ed0f	SUCCESS	90040	63	22	30	38

Folio Development Teams

[Poppy] List App on a single tenant and R/W split enabled and multiple lists

Overview

Summary

Test runs

Results

CPU Utilization

LA_test_1

LA_test_5

LA_test_10

Memory Utilization

LA_test_1.

LA_test_5

LA_test_10

DB CPU Utilization

DB Connections

DB Load

Appendix

Infrastructure

Methodology

Additional Screenshots of graphs or charts (Optional)