Overview

PERF-932 - Getting issue details... STATUS was created to re-run RTAC performance test in the Quesnelia environment which will verify performance issues reported by PERF-895 - Getting issue details... STATUS

We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.

Summary

On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
On average mod-rtac POST_/rtac-batch API call takes 487 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
On running mod-inventory-storage POST_inventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle.

Test Runs

Test	Concurrent Virtual Users	Duration	Holdings per instance	API call
1.	1	30 min	greater than 300	mod-rtac POST https://[hostName]/rtac-batch
2.	1	30 min	greater than 300	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
3.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
4.	1	30 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
5.	2	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
6.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-rtac POST https://[hostName]/rtac-batch
7.	15	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings
8.	20	15 min	50 instances with a mix of 3,4,5,6,7 holdings per instance	mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings

Results

mod-rtac

A mod-rtac level request to get all open loans for greater than 300 holdings per instance

POST /rtac-batch for 30 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
1 vUser	1	333	790 ms	854 ms	529 ms	+261 ms
1 vUser	50	mix of 3,4,5,6,7	487 ms	516 ms	2.57 seconds	-2.1 sec
1 vUser	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	2659 ms	2933 ms	-	-
1 vUser	1	>=10 from PERF-895 - Getting issue details... STATUS	82 ms	86 ms	-	-

This test was run to exaggerate with 20 virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.

POST /rtac-batch for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
2 vUsers	50	mix of 3,4,5,6,7	538 ms	627 ms	2.97 seconds	-2.4 sec
20 vUsers	50	mix of 3,4,5,6,7	3622 ms	4194 ms	21.75 seconds	-18.1 sec

No errors for mod-rtac were found in CloudWatch logs

mod-rtac calls inventory-hierarchy. API tests were run for 15 virtual users and 20 virtual users. No errors were reported.

POST inventory-hierarchy/items-and-holdings for 15 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
15 vUsers	50	mix of 3,4,5,6,7	371 ms	513 ms	7.11 seconds	-6.7 sec
20 vUsers	50	mix of 3,4,5,6,7	500 ms	704 ms	9.34 seconds	-8.8 sec

edge-rtac

An edge-rtac level request to get all open loans for greater than 300 holdings per instance

GET /rtac for 30 minutes	Total Instances	Total Holdings	Average	95th %tile	Average (Honeysuckle)	Delta
1 vUser	1	333	800 ms	972 ms	721 ms	+79 ms
1 vUser	50	mix of 3,4,5,6,7	807 ms	1349 ms	2.58 seconds	-1.7 sec
1 vUser	50	mix of 11,12	5796 ms	10997 ms	-	-
2 vUsers	50	mix of 11,12	6904 ms	14619 ms	-	-
20 vUsers	50	mix of 11,12	34725 ms	57423 ms	-	-
1 vUser	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	4887 ms	5165 ms	-	-
50 vUsers	30	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	63966 ms	10123 ms	-	-
1 vUser	1	>=10 (one has 5k) from PERF-895 - Getting issue details... STATUS	239 ms	412 ms	-	-

Slow Queries

1 instance with 333 holdings

	Average Time, ms	Module	Query
1 vUser	119 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

50 instances with a mix of 3,4,5,6,7 holdings per instance

	Average Time, ms	Module	Query
20 vUsers	1057 ms	mod_inventory_storage	select * from get_items_and_holdings_view($1,$2)

RDS CPU Utilization

CPU is around 99.5% for /inventory-hierarchy. For 1 user was about 18%

Service Memory Utilization

Service CPU Utilization

Appendix

14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge database instance: Writer instance
MSK tenant [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt

mod-rtac	5	`mod-rtac:3.6.0`	2	1024	896	128	768	88	128
mod-authtoken	6	`mod-authtoken:2.15.1`	2	1440	1152	512	922	88	128
mod-permissions	10	`mod-permissions:6.5.0`	2	1684	1544	512	1024	384	512
mod-inventory-storage	5	`mod-inventory-storage:27.1.0`	2	4096	3690	2048	3076	384	512
mod-circulation-storage	5	`mod-circulation-storage:17.2.0`	2	2880	2592	1536	1814	384	512
mod-inventory	5	`mod-inventory:20.2.0`	2	2880	2592	1024	1814	384	512
mod-circulation	5	`mod-circulation:24.2.0`	2	2880	2592	1536	1814	384	512
edge-rtac	5	`edge-rtac:2.7.1`	2	1024	896	128	768	88	128
nginx-okapi	5	`nginx-okapi:2023.06.14`	2	1024	896	128	0	0	0
okapi	5	`okapi:5.3.0`	3	1684	1440	1024	922	384	512

Methodology/Approach

Run the test from the script (located in the "workflows-scripts/RTAC/rtac-all-for-regress" folder on GitHub) one by one. 8 tests as described in Test Runs.

There are pictures of Explain Analyze of long-running query:

Browser not supported