PTF -RTAC Workflow Test Report (Quesnelia)
- Olga Kondratenko
WIP
Work in progress...
Overview
PERF-932 - Getting issue details... STATUS was created to re-run RTAC performance test in the Quesnelia environment which will verify performance issues reported by PERF-895 - Getting issue details... STATUS
We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.
Summary
- On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
- On average mod-rtac POST_/rtac-batch API call takes 487 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
- On running mod-inventory-storage POST_inventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle.
Test Runs
Test | Concurrent Virtual Users | Duration | Holdings per instance | API call |
1. | 1 | 30 min | greater than 300 | mod-rtac POST https://[hostName]/rtac-batch |
| 2. | 1 | 30 min | greater than 300 | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 3. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 4. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 5. | 2 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 6. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 7. | 15 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
| 8. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
Results
mod-rtac
A mod-rtac level request to get all open loans for greater than 300 holdings per instance
| POST /rtac-batch for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 1 vUser | 1 | 333 | 790 ms | 854 ms | 529 ms | +261 ms |
| 1 vUser | 50 | mix of 3,4,5,6,7 | 487 ms | 516 ms | 2.57 seconds | -2.1 sec |
| 1 vUser | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 2659 ms | 2933 ms | - | - |
| 1 vUser | 1 | >=10 from PERF-895 - Getting issue details... STATUS | 82 ms | 86 ms | - | - |
This test was run to exaggerate with 20 virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.
| POST /rtac-batch for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 2 vUsers | 50 | mix of 3,4,5,6,7 | 538 ms | 627 ms | 2.97 seconds | -2.4 sec |
| 20 vUsers | 50 | mix of 3,4,5,6,7 | 3622 ms | 4194 ms | 21.75 seconds | -18.1 sec |
No errors for mod-rtac were found in CloudWatch logs
mod-rtac calls inventory-hierarchy. API tests were run for 15 virtual users and 20 virtual users. No errors were reported.
| POST inventory-hierarchy/items-and-holdings for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 15 vUsers | 50 | mix of 3,4,5,6,7 | 371 ms | 513 ms | 7.11 seconds | -6.7 sec |
| 20 vUsers | 50 | mix of 3,4,5,6,7 | 500 ms | 704 ms | 9.34 seconds | -8.8 sec |
edge-rtac
An edge-rtac level request to get all open loans for greater than 300 holdings per instance
| GET /rtac for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 1 vUser | 1 | 333 | 800 ms | 972 ms | 721 ms | +79 ms |
| 1 vUser | 50 | mix of 3,4,5,6,7 | 807 ms | 1349 ms | 2.58 seconds | -1.7 sec |
| 1 vUser | 50 | mix of 11,12 | 5796 ms | 10997 ms | - | - |
| 2 vUsers | 50 | mix of 11,12 | 6904 ms | 14619 ms | - | - |
| 20 vUsers | 50 | mix of 11,12 | 34725 ms | 57423 ms | - | - |
| 1 vUser | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 4887 ms | 5165 ms | - | - |
| 50 vUsers | 30 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 63966 ms | 10123 ms | - | - |
| 1 vUser | 1 | >=10 (one has 5k) from PERF-895 - Getting issue details... STATUS | 239 ms | 412 ms | - | - |
Slow Queries
1 instance with 333 holdings
Average Time, ms | Module | Query | |
|---|---|---|---|
| 1 vUser | 119 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
50 instances with a mix of 3,4,5,6,7 holdings per instance
Average Time, ms | Module | Query | |
|---|---|---|---|
| 20 vUsers | 1057 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
RDS CPU Utilization
CPU is around 99.5% for /inventory-hierarchy. For 1 user was about 18%
Service Memory Utilization
Service CPU Utilization
Appendix
- 14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 1 instance of db.r6.xlarge database instance: Writer instance
- MSK tenant [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
| qcp1-pvt | |||||||||
| mod-rtac | 5 | | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| mod-authtoken | 6 | | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
| mod-permissions | 10 | | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 |
| mod-inventory-storage | 5 | | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
| mod-circulation-storage | 5 | | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| mod-inventory | 5 | | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
| mod-circulation | 5 | | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| edge-rtac | 5 | | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| nginx-okapi | 5 | | 2 | 1024 | 896 | 128 | 0 | 0 | 0 |
| okapi | 5 | | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
Methodology/Approach
Run the test from the script (located in the "workflows-scripts/RTAC/rtac-all-for-regress" folder on GitHub) one by one. 8 tests as described in Test Runs.
There are pictures of Explain Analyze of long-running query:
