WIP
Work in progress...
Overview
PERF-932 - Getting issue details... STATUS was created to re-run RTAC performance test in the Honeysuckle environment which will verify performance issues reported by PERF-895 - Getting issue details... STATUS
We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.
Summary
- On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 721.40 ms for 1 instance with 300+ holdings. This is an improvement from 17.5 seconds in Fameflower.
- On average mod-rtac POST_/rtac-batch API call takes 529 ms for 1 instance with 300+ holdings. This is an improvement from 17 seconds in Fameflower.
- On running mod-rtac POST_/rtac-batch API for more than 5 users for 15 minutes test run, mapping errors were seen. Please see MODRTAC-53 in recommended improvements below.
Test Runs
Test | Virtual Users | Duration | Holdings per instance | API call |
1. | 1 | 30 min | greater than 300 | mod-rtac POST https://[hostName]/rtac-batch |
| 2. | 1 | 30 min | greater than 300 | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 3. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 4. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 5. | 2 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 6. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 7. | 15 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
| 8. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
Results
mod-rtac
A mod-rtac level request to get all open loans for greater than 300 holdings per instance
| POST /rtac-batch for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 1 user | 1 | 333 | 790 ms | 854 ms | 529 ms | |
| 1 user | 50 | mix of 3,4,5,6,7 | 487 ms | 516 ms | 2.57 seconds | |
| 1 user | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 2659 ms | 2933 ms | - | - |
| 1 user | 1 | >=10 from PERF-895 - Getting issue details... STATUS | 82 ms | 86 ms | - | - |
This test was run to exaggerate with 20 users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.
| POST /rtac-batch for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 2 users | 50 | mix of 3,4,5,6,7 | 538 ms | 627 ms | 2.97 seconds | |
| 20 users | 50 | mix of 3,4,5,6,7 | 3622 ms | 4194 ms | 21.75 seconds |
No errors for mod-rtac were found in CloudWatch logs
mod-rtac calls inventory-hierarchy. To investigate the error for mod-rtac for concurrent users(2 or more), POST inventory-hierarchy/items-and-holdings API tests were run for 15 users and 20 users. No errors were reported.
| POST inventory-hierarchy/items-and-holdings for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 15 users | 50 | mix of 3,4,5,6,7 | 371 ms | 513 ms | 7.11 seconds | |
| 20 users | 50 | mix of 3,4,5,6,7 | 500 ms | 704 ms | 9.34 seconds |
edge-rtac
An edge-rtac level request to get all open loans for greater than 300 holdings per instance
| GET /rtac for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 1 user | 1 | 333 | 800 ms | 972 ms | 721 ms | |
| 1 user | 50 | mix of 3,4,5,6,7 | 807 ms | 1349 ms | 2.58 seconds | |
| 1 user | 50 | mix of 11,12 | 5796 ms | 10997 ms | - | - |
| 2 users | 50 | mix of 11,12 | 6904 ms | 14619 ms | - | - |
| 20 users | 50 | mix of 11,12 | 34725 ms | 57423 ms | - | - |
| 1 user | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 4887 ms | 5165 ms | - | - |
| 50 users | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 63966 ms | 10123 ms | - | - |
| 1 user | 1 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 239 ms | 412 ms | - | - |
Slow Queries
1 instance with 333 holdings
Average Time, ms | Module | Query | |
|---|---|---|---|
| 1 user | 119 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
50 instances with a mix of 3,4,5,6,7 holdings per instance
Average Time, ms | Module | Query | |
|---|---|---|---|
| 20 users | 1057 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
RDS CPU Utilization
1 user 1 instance with 333 holdings. CPU is around 24% which is normal.
.
1 user 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is around 27% which is normal.
.
15 users, 20 users 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is close to 100% which is high because DB was fired with 15, 20 different users. This run involved 504 errors from mod-rtac. These many numbers of concurrent users are rare and will not happen in production.
Service Memory Utilization
15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. Service memory Utilization is relatively high for mod-rtac compared to other modules in the graph but this is rare. This use case was run to recreate the issue reported in MODRTAC-53. 2 vertical lines represent the time range for the test run.
.
Service CPU Utilization
15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU Utilization is relatively high but this is rare. This use case was run to recreate the issue reported in MODRTAC-53
.
Appendix
Backend:
- mod-circulation-19.2.3
- mod-circulation-storage-12.1.2
- mod-inventory-storage-19.4.3
- mod-inventory-16.1.0
- okapi-4.2.2
- mod-authtoken-2.6.0
- mod-permissions-5.12.2
Environment:
- 63 back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 10 m5.large EC2 instances
- 2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)