PERF-117 - Getting issue details... STATUS

WIP

Work in progress...

Overview

PERF-932 - Getting issue details... STATUS was created to re-run RTAC performance test in the Honeysuckle environment which will verify performance issues reported by PERF-895 - Getting issue details... STATUS

We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.

Summary

On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 721.40 ms for 1 instance with 300+ holdings. This is an improvement from 17.5 seconds in Fameflower.
On average mod-rtac POST_/rtac-batch API call takes 529 ms for 1 instance with 300+ holdings. This is an improvement from 17 seconds in Fameflower.
On running mod-rtac POST_/rtac-batch API for more than 5 users for 15 minutes test run, mapping errors were seen. Please see MODRTAC-53 in recommended improvements below.

Test Runs

Test	Virtual Users	Duration	Holdings per instance	API call
1.	1	30 min	greater than 300	mod-rtac POST https://[hostName]/rtac-batch
2.	1	30 min	greater than 300	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
3.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
4.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
5.	2	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
6.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
7.	15	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings
8.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings

Results

mod-rtac

A mod-rtac level request to get all open loans for greater than 300 holdings per instance

POST /rtac-batch for 30 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
1 user	1	333	790 ms	854 ms	529 ms
1 user	50	mix of 3,4,5,6,7	487 ms	516 ms	2.57 seconds
1 user	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	2659 ms	2933 ms	-	-
1 user	1	>=10 from PERF-895 - Getting issue details... STATUS	82 ms	86 ms	-	-

This test was run to exaggerate with 20 users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.

POST /rtac-batch for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
2 users	50	mix of 3,4,5,6,7	538 ms	627 ms	2.97 seconds
20 users	50	mix of 3,4,5,6,7	3622 ms	4194 ms	21.75 seconds

No errors for mod-rtac were found in CloudWatch logs

mod-rtac calls inventory-hierarchy. To investigate the error for mod-rtac for concurrent users(2 or more), POST inventory-hierarchy/items-and-holdings API tests were run for 15 users and 20 users. No errors were reported.

POST inventory-hierarchy/items-and-holdings for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
15 users	50	mix of 3,4,5,6,7	371 ms	513 ms	7.11 seconds
20 users	50	mix of 3,4,5,6,7	500 ms	704 ms	9.34 seconds

edge-rtac

An edge-rtac level request to get all open loans for greater than 300 holdings per instance

GET /rtac for 30 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
1 user	1	333	800 ms	972 ms	721 ms
1 user	50	mix of 3,4,5,6,7	807 ms	1349 ms	2.58 seconds
1 user	50	mix of 11,12	5796 ms	10997 ms	-	-
2 users	50	mix of 11,12	6904 ms	14619 ms	-	-
20 users	50	mix of 11,12	34725 ms	57423 ms	-	-
1 user	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	4887 ms	5165 ms	-	-
50 users	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	63966 ms	10123 ms	-	-
1 user	1	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	239 ms	412 ms	-	-

Slow Queries

1 instance with 333 holdings

	Average Time, ms	Module	Query
1 user	119 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

50 instances with a mix of 3,4,5,6,7 holdings per instance

	Average Time, ms	Module	Query
20 users	1057 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

RDS CPU Utilization

1 user 1 instance with 333 holdings. CPU is around 24% which is normal.

.

1 user 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is around 27% which is normal.

.

15 users, 20 users 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is close to 100% which is high because DB was fired with 15, 20 different users. This run involved 504 errors from mod-rtac. These many numbers of concurrent users are rare and will not happen in production.

Service Memory Utilization

15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. Service memory Utilization is relatively high for mod-rtac compared to other modules in the graph but this is rare. This use case was run to recreate the issue reported in MODRTAC-53. 2 vertical lines represent the time range for the test run.

.

Service CPU Utilization

15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU Utilization is relatively high but this is rare. This use case was run to recreate the issue reported in MODRTAC-53

.

Appendix

Backend:

mod-circulation-19.2.3
mod-circulation-storage-12.1.2
mod-inventory-storage-19.4.3
mod-inventory-16.1.0
okapi-4.2.2
mod-authtoken-2.6.0
mod-permissions-5.12.2

Environment:

63 back-end modules deployed in 110 ECS services
3 okapi ECS services
10 m5.large EC2 instances
2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)

PTF -RTAC Workflow Test Report (Quesnelia)