Table of Contents

Info

title	WIP

Work in progress...

Overview

Jira Legacy

01505d01

server	System JIRAJira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-117

Table of Contents

Info

title	WIP

Work in progress...

Overview

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-932

was created to re-

932

was created to re-run RTAC performance test in the Honeysuckle the Quesnelia environment which will verify performance issues reported by

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

...

On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 721.40 ms for 1 instance with 300+ 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 172.5 6 seconds in Fameflower Honeysuckle.
On average mod-rtac POST_/rtac-batch API call takes 529 487 ms for 1 instance with 300+ 50 instances with 3,4,5,6,7 holdings. This is an improvement from 17 2.6 seconds in FameflowerHoneysuckle.
On running mod-rtac running mod-inventory-storage POST_/rtac-batch API for more than 5 users for 15 minutes test run, mapping errors were seen. Please see MODRTAC-53 in recommended improvements belowinventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle.

Test Runs

Test	Concurrent Virtual Users	Duration	Holdings per instance	API call
1.	1	30 min	greater than 300	mod-rtac POST https://[hostName]/rtac-batch
2.	1	30 min	greater than 300	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
3.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
4.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
5.	2	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
6.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
7.	15	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings
8.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings

Results

mod-rtac

A mod-rtac level request to get all open loans for greater than 300 holdings per instance

1 user

POST /rtac-batch for 30 minutes

Total Instances

Total Holdings

Average

95th %tile

Average (Honeysuckle)

Delta

1 uservUser

1

333

790 ms

854 ms

529 ms

+261 ms

1 user1 vUser

50

mix of 3,4,5,6,7

487 ms

516 ms

2.57 seconds

-2.1 sec

1 vUser

30

>=10 (one has 5k) from

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

2659 ms

2933 ms

-

1 user1 vUser

1

>=10 from

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

82 ms

86 ms

-

...

This test was run to exaggerate with 20 users virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.

20 users

POST /rtac-batch for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
2 users2 vUsers	50	mix of 3,4,5,6,7	538 ms	627 ms	2.97 seconds	-2.4 sec
20 vUsers	50	mix of 3,4,5,6,7	3622 ms	4194 ms	21.75 seconds	-18.1 sec

No errors for mod-rtac were found in CloudWatch logs

mod-rtac calls inventory-hierarchy. To investigate the error for mod-rtac for concurrent users(2 or more), POST inventory-hierarchy/items-and-holdings API tests were run for 15 users and 20 users. No errors were API tests were run for 15 virtual users and 20 virtual users. No errors were reported.

20 users

POST inventory-hierarchy/items-and-holdings for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
15 users15 vUsers	50	mix of 3,4,5,6,7	371 ms	513 ms	7.11 seconds	-6.7 sec
20 vUsers	50	mix of 3,4,5,6,7	500 ms	704 ms	9.34 seconds	-8.8 sec

edge-rtac

An edge-rtac level request to get all open loans for greater than 300 holdings per instance

user

GET /rtac for 30 minutes

Total Instances

Total Holdings

Average

95th %tile

Average (Honeysuckle)

Delta

1 user1 vUser

1

333

800 ms

972 ms

721 ms

+79 ms

1 user1 vUser

50

mix of 3,4,5,6,7

807 ms

1349 ms

2.58 seconds

-1

.7 sec

1 vUser

50

mix of 11,12

5796 ms

10997 ms

-

2 users2 vUsers

50

mix of 11,12

6904 ms

14619 ms

-

20 users20 vUsers

50

mix of 11,12

34725 ms

57423 ms

-

1 user1 vUser

30

>=10 (one has 5k) from

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

4887 ms

5165 ms

-

50 users50 vUsers

30

>=10 (one has 5k) from

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

63966 ms

10123 ms

-

1 user1 vUser

1

>=10 (one has 5k) from

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-895

239 ms

412 ms

-

...

	Average Time, ms	Module	Query
1 user1 vUser	119 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

50 instances with a mix of 3,4,5,6,7 holdings per instance

	Average Time, ms	Module	Query
20 users20 vUsers	1057 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

RDS CPU Utilization

1 user 1 instance with 333 holdings. CPU is around 24% which is normal.

.

1 user 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is around 27% which is normal.

.

15 users, 20 users 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is close to 100% which is high because DB was fired with 15, 20 different users. This run involved 504 errors from mod-rtac. These many numbers of concurrent users are rare and will not happen in production.

Service Memory Utilization

15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. Service memory Utilization is relatively high for mod-rtac compared to other modules in the graph but this is rare. This use case was run to recreate the issue reported in MODRTAC-53. 2 vertical lines represent the time range for the test run.

.

Service CPU Utilization

15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU Utilization is relatively high but this is rare. This use case was run to recreate the issue reported in MODRTAC-53

.

Appendix

Backend:

mod-circulation-19.2.3
mod-circulation-storage-12.1.2
mod-inventory-storage-19.4.3
mod-inventory-16.1.0
okapi-4.2.2
mod-authtoken-2.6.0
mod-permissions-5.12.2

Environment:

...

99.5% for /inventory-hierarchy. For 1 user was about 18%

Image Added

Service Memory Utilization

Image Added

Service CPU Utilization

Image Added

Appendix

14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge database instance: Writer instance
MSK tenant [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt

mod-rtac	5	`mod-rtac:3.6.0`	2	1024	896	128	768	88	128
mod-authtoken	6	`mod-authtoken:2.15.1`	2	1440	1152	512	922	88	128
mod-permissions	10	`mod-permissions:6.5.0`	2	1684	1544	512	1024	384	512
mod-inventory-storage	5	`mod-inventory-storage:27.1.0`	2	4096	3690	2048	3076	384	512
mod-circulation-storage	5	`mod-circulation-storage:17.2.0`	2	2880	2592	1536	1814	384	512
mod-inventory	5	`mod-inventory:20.2.0`	2	2880	2592	1024	1814	384	512
mod-circulation	5	`mod-circulation:24.2.0`	2	2880	2592	1536	1814	384	512
edge-rtac	5	`edge-rtac:2.7.1`	2	1024	896	128	768	88	128
nginx-okapi	5	`nginx-okapi:2023.06.14`	2	1024	896	128	0	0	0
okapi	5	`okapi:5.3.0`	3	1684	1440	1024	922	384	512

Methodology/Approach

Run the test from the script (located in the "workflows-scripts/RTAC/rtac-all-for-regress" folder on GitHub) one by one. 8 tests as described in Test Runs.

There are pictures of Explain Analyze of long-running query:

Image Added

Versions Compared

Old Version 1

New Version Current

Key

Overview

Overview

Test Runs

Results

mod-rtac

edge-rtac

RDS CPU Utilization

Service Memory Utilization

.

Service CPU Utilization

.

Appendix

Service Memory Utilization

Image Added

Service CPU Utilization

Image Added

Appendix

Methodology/Approach

Page Comparison

Versions Compared

Old Version 1

New Version Current

Key

Overview

Overview

Test Runs

Results

mod-rtac

edge-rtac

RDS CPU Utilization

Service Memory Utilization

.

Service CPU Utilization

.

Appendix

Service Memory Utilization

Image Added

Service CPU Utilization

Image Added

Appendix

Methodology/Approach