PTF -RTAC Workflow Test Report (Quesnelia)
WIP
Work in progress...
Overview
- PERF-932Getting issue details... STATUS was created to re-run RTAC performance test in the Quesnelia environment which will verify performance issues reported by - PERF-895Getting issue details... STATUS
We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.
Summary
- On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
- On average mod-rtac POST_/rtac-batch API call takes 487 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
- On running mod-inventory-storage POST_inventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle.
Test Runs
Test | Concurrent Virtual Users | Duration | Holdings per instance | API call |
1. | 1 | 30 min | greater than 300 | mod-rtac POST https://[hostName]/rtac-batch |
2. | 1 | 30 min | greater than 300 | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
3. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
4. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
5. | 2 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
6. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
7. | 15 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
8. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
Results
mod-rtac
A mod-rtac level request to get all open loans for greater than 300 holdings per instance
This test was run to exaggerate with 20 virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.
POST /rtac-batch for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
2 vUsers | 50 | mix of 3,4,5,6,7 | 538 ms | 627 ms | 2.97 seconds | -2.4 sec |
20 vUsers | 50 | mix of 3,4,5,6,7 | 3622 ms | 4194 ms | 21.75 seconds | -18.1 sec |
No errors for mod-rtac were found in CloudWatch logs
mod-rtac calls inventory-hierarchy. API tests were run for 15 virtual users and 20 virtual users. No errors were reported.
POST inventory-hierarchy/items-and-holdings for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
15 vUsers | 50 | mix of 3,4,5,6,7 | 371 ms | 513 ms | 7.11 seconds | -6.7 sec |
20 vUsers | 50 | mix of 3,4,5,6,7 | 500 ms | 704 ms | 9.34 seconds | -8.8 sec |
edge-rtac
An edge-rtac level request to get all open loans for greater than 300 holdings per instance
Slow Queries
1 instance with 333 holdings
Average Time, ms | Module | Query | |
---|---|---|---|
1 vUser | 119 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
50 instances with a mix of 3,4,5,6,7 holdings per instance
Average Time, ms | Module | Query | |
---|---|---|---|
20 vUsers | 1057 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
RDS CPU Utilization
CPU is around 99.5% for /inventory-hierarchy. For 1 user was about 18%
Service Memory Utilization
Service CPU Utilization
Appendix
- 14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 1 instance of db.r6.xlarge database instance: Writer instance
- MSK tenant [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
qcp1-pvt | |||||||||
mod-rtac | 5 |
| 2 |