...

Table of Contents
Overview

This document contains the results of testing List App refreshing of 200k records with multiple parallel workflows (Check-in/Check-out and Data import) in the Poppy release. The goal of testing is to assess the performance of mod-lists along with other workflows (CICO and DI) and to measure mod-lists's impact on other modules

Ticket:

Jira Legacy

server	System JiraJIRA
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-668

Jira Legacy

server	System JiraJIRA
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-691

Summary

ListApp refreshing duration is about 8.5 min when testing separately from other workflows. With Data Import and Check-in/Check-out scenarios added ListApp duration has +108% increase and reaches 17.7 min. It's above the expected value of 10 minutes.
Data Import and Check-in/Check-out response times have increase about 10-20% with ListApp scenario added.
According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.
RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

Test runs

Query used in lists - "Item status == Checked out". List refresh result is 200K records.

Scenario

Data quantity

List App * + Check-in/Check-out + Data Import Create**

ListApp - 10 users

CICO - 20 users

DI - 25k

Check-in/Check-out + Data import**

CICO - 20 users

DI - 25k

List App

ListApp - 10 users

*200k records list refresh for each user.**PTF Create 2 PTF Create 2 job was used for DI scneario

Results

Transaction	Duration			Degradation
Transaction	[Orchid] Testing separately	[Poppy] Testing separately (ListApp and DI+CICO)	[Poppy] Testing in parallel	Degradation
ListApp refresh, avg	from 9 min 40 sec to 10 min 40 sec	8.5 min	17.7 min	+108%
Check-in, avg		1.173 sec	1.290 sec	+9.9%
Check-out, avg		1.648 sec	1.948 sec	+18.2%
DI Create 25k		18 min	22 min	+22%

Grafana results

DI + CICO

ListApp + DI + CICO

Instance CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.

ListApp

Image RemovedImage Added

DI + CICO

Image RemovedImage Added

ListApp + DI + CICO

Image RemovedImage Added

Memory Utilization

No memory leaks were found during the tests.

ListApp

Image RemovedImage Added

DI + CICO

Image RemovedImage Added

ListApp + DI + CICO

Image RemovedImage Added

DB CPU Utilization

RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

DI + CICO

ListApp + DI + CICO

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Module	Task Def. Revision	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage:3.0.0-SNAPSHOT.133	9	2	4920	4472	1024	3960	512	512	false
mod-finance-storage:8.5.0-SNAPSHOT.150	8	2	1024	896	1024	700	88	128	false
mod-agreements:6.0.0-SNAPSHOT.161	8	2	1592	1488	128	968	384	512	false
mod-ebsconet:2.1.0-SNAPSHOT.64	8	2	1248	1024	128	700	128	256	false
mod-organizations:1.8.0-SNAPSHOT.66	8	2	1024	896	128	700	88	128	false
mod-settings:1.1.0-SNAPSHOT.49	8	2	1024	896	200	0	0	0	false
edge-dematic:3.0.0-SNAPSHOT.114	8	1	1024	896	128	0	0	0	false
mod-data-import:2.8.0-SNAPSHOT.148	9	1	2048	1844	256	1292	384	512	false
mod-search:3.0.0-SNAPSHOT.151	16	2	2592	2480	2048	1440	512	1024	false
mod-authtoken:2.14.0-SNAPSHOT.263	8	2	1440	1152	512	922	88	128	false
mod-configuration:5.9.2-SNAPSHOT.336	8	2	1024	896	128	768	88	128	false
mod-orders-storage:13.6.0-SNAPSHOT.204	8	2	1024	896	512	700	88	128	false
mod-fqm-manager:1.0.0-SNAPSHOT.929	3	2	3000	2600	128	2048	384	512	true
mod-bulk-operations:2.0.0-SNAPSHOT.53	6	2	3072	2600	1024	1536	384	512	false
mod-graphql:1.11.0.364	12	2	1024	896	128	768	88	128	false
mod-finance:4.8.0-SNAPSHOT.141	8	2	1024	896	128	700	88	128	false
mod-erm-usage:4.6.0-SNAPSHOT.93	11	2	1024	896	128	768	88	128	false
mod-copycat:1.5.0-SNAPSHOT.159	8	2	1024	896	128	768	88	128	false
mod-lists:1.0.0-SNAPSHOT.1199	3	2	3000	2600	128	2048	384	512	false
mod-permissions:6.4.0-SNAPSHOT.347	28	2	1684	1544	512	1024	384	512	false
mod-orders:12.7.0-SNAPSHOT.315	8	2	2048	1440	1024	1024	384	512	false
mod-users-bl:7.6.0-SNAPSHOT.260	8	2	1440	1152	512	922	88	128	false
mod-inventory-storage:26.1.0-SNAPSHOT.696	9	2	4096	3690	2048	3076	384	512	false
mod-invoice:5.7.0-SNAPSHOT.164	8	2	1440	1152	512	922	88	128	false
mod-user-import:3.8.0-SNAPSHOT.183	8	2	1024	896	128	768	88	128	false
edge-oai-pmh:2.7.0-SNAPSHOT.141	6	2	1512	1360	1024	1440	384	512	false
mod-data-export-worker:3.1.0-SNAPSHOT.174	8	2	3072	2800	1024	2048	384	512	false
mod-circulation-storage:16.1.0-SNAPSHOT.356	9	2	2880	2592	1536	1814	384	512	false
mod-source-record-storage:5.7.0-SNAPSHOT.308	9	2	5600	5000	2048	3500	384	512	false
mod-inventory:20.1.0-SNAPSHOT.446	8	2	2880	2592	1024	1814	384	512	false
mod-email:1.15.4-SNAPSHOT.113	8	2	1024	896	128	768	88	128	false
mod-di-converter-storage:2.1.0-SNAPSHOT.64	12	2	1024	896	128	768	88	128	false
mod-circulation:24.0.0-SNAPSHOT.601	9	2	2880	2592	1536	768	88	128	false
mod-pubsub:2.10.0-SNAPSHOT.167	8	2	1536	1440	1024	922	384	512	false
edge-orders:2.9.0-SNAPSHOT.110	8	2	1024	896	128	768	88	128	false
mod-users:19.2.0-SNAPSHOT.638	18	2	1024	896	128	768	88	128	false
folio/mod-patron-blocks:1.9.0-SNAPSHOT.119	8	2	1024	896	1024	768	88	128	false
folio/mod-audit:2.7.1-SNAPSHOT.189	8	2	1024	896	128	768	88	128	false
folio/mod-source-record-manager:3.7.0-SNAPSHOT.240	8	2	5600	5000	2048	3500	384	512	false
folio/nginx-edge:2023.06.14	6	2	1024	896	128	0	0	0	false
folio/mod-quick-marc:5.0.0-SNAPSHOT.114	8	1	2288	2176	128	1664	384	512	false
folio/nginx-okapi:2023.06.14	6	2	1024	896	128	0	0	0	false
folio/okapi:5.0.1	7	3	1684	1440	1024	922	384	512	false
folio/mod-feesfines:18.3.0-SNAPSHOT.172	9	2	1024	896	128	768	88	128	false
folio/mod-invoice-storage:5.7.0-SNAPSHOT.97	8	2	1872	1536	1024	1024	384	512	false
folio/mod-service-interaction:3.0.0-SNAPSHOT.87	8	2	2048	1844	256	1290	384	512	false
folio/mod-data-export:4.8.0-SNAPSHOT.181	9	1	1024	896	1024	768	88	128	false
folio/mod-patron:6.0.0-SNAPSHOT.138	8	2	1024	896	128	768	88	128	false
folio/mod-oai-pmh:3.12.0-SNAPSHOT.362	3	2	4096	3690	2048	3076	384	512	false
mod-organizations-storage:4.6.0-SNAPSHOT.106	8	2	1024	896	128	700	88	128	false
mod-data-export-spring:3.0.0-SNAPSHOT.129	10	1	2048	1844	256	1536	384	512	false
folio/mod-login:7.10.0-SNAPSHOT.304	8	2	1440	1298	1024	768	384	512	false
folio/pub-okapi:2023.06.14	6	2	1024	896	128	768	0	0	false

Additional info

Grafana graphs:

List App + Check-in/Check-out + Data Import Create

...

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&from=1696927212822&to=1696931260640&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Methodology

...

1.Prepare data for ListApp: check-out 200K items with CICO JMeter script.

2. Disable timers before running the tests:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers

3. Conduct test with ListApp scenario

4. Conduct test with ListApp scenario + CICO + DI

5. Conduct test withCICO + DI scenarios

6. Compare the results

7. Enable timers:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"30"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"35"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"20"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"60"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers

Versions Compared

Old Version 30

New Version Current

Key

Table of Contents
Overview

Summary

Test runs

Results

Grafana results

DI + CICO

ListApp + DI + CICO

Instance CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Memory Utilization

ListApp

DI + CICO

ListApp + DI + CICO

DB CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

DI + CICO

ListApp + DI + CICO

Appendix

Infrastructure

Additional info

Methodology

Name	API Name	Memory GIB	vCPUs	max_connections
R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

Page Comparison

Versions Compared

Old Version 30

New Version Current

Key

<span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-0">[data-colorid=</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-4">f43xaay5vq</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-5">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-6">f43xaay5vq</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-7">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-8">i3ml2hwxm6</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-9">]{color:</span>

Summary

Test runs

Results

Grafana results

DI + CICO

ListApp + DI + CICO

Instance CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Memory Utilization

ListApp

DI + CICO

ListApp + DI + CICO

DB CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

DI + CICO

ListApp + DI + CICO

Appendix

Infrastructure

Additional info

Methodology