| Table of Contents |
|---|
| Info | ||
|---|---|---|
| ||
Work in progress... |
...
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
...
- On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 721.40 ms for 1 instance with 300+ 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 172.5 6 seconds in Fameflower Honeysuckle.
- On average mod-rtac POST_/rtac-batch API call takes 529 487 ms for 1 instance with 300+ 50 instances with 3,4,5,6,7 holdings. This is an improvement from 17 2.6 seconds in FameflowerHoneysuckle.
- On running mod-rtac POST_/rtac-batch API for more than 5 users for 15 minutes test run, mapping errors were seen. Please see MODRTAC-53 in recommended improvements belowrunning mod-inventory-storage POST_inventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle.
Test Runs
Test | Concurrent Virtual Users | Duration | Holdings per instance | API call |
1. | 1 | 30 min | greater than 300 | mod-rtac POST https://[hostName]/rtac-batch |
| 2. | 1 | 30 min | greater than 300 | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 3. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true |
| 4. | 1 | 30 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 5. | 2 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 6. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-rtac POST https://[hostName]/rtac-batch |
| 7. | 15 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
| 8. | 20 | 15 min | 50 instances with a mix of 3,4,5,6,7 holdings per instance | mod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings |
Results
mod-rtac
A mod-rtac level request to get all open loans for greater than 300 holdings per instance
| POST /rtac-batch for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta | ||||||||
| 1 uservUser | 1 | 333 | 790 ms | 854 ms | 529 ms | +261 ms | ||||||||
| 1 user1 vUser | 50 | mix of 3,4,5,6,7 | 487 ms | 516 ms | 2.57 seconds | 1 user-2.1 sec | ||||||||
| 1 vUser | 30 | >=10 (one has 5k) from
| 2659 ms | 2933 ms | - | - | ||||||||
| 1 user1 vUser | 1 | >=10 from
| 82 ms | 86 ms | - | - |
...
This test was run to exaggerate with 20 users virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.
| POST /rtac-batch for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 2 users2 vUsers | 50 | mix of 3,4,5,6,7 | 538 ms | 627 ms | 2.97 seconds | 20 users-2.4 sec |
| 20 vUsers | 50 | mix of 3,4,5,6,7 | 3622 ms | 4194 ms | 21.75 seconds | -18.1 sec |
No errors for mod-rtac were found in CloudWatch logs
mod-rtac calls inventory-hierarchy. To investigate the error for mod-rtac for concurrent users(2 or more), POST inventory-hierarchy/items-and-holdings API tests were run for 15 users and 20 API tests were run for 15 virtual users and 20 virtual users. No errors were reported.
| POST inventory-hierarchy/items-and-holdings for 15 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta |
| 15 users15 vUsers | 50 | mix of 3,4,5,6,7 | 371 ms | 513 ms | 7.11 seconds | 20 users-6.7 sec |
| 20 vUsers | 50 | mix of 3,4,5,6,7 | 500 ms | 704 ms | 9.34 seconds | -8.8 sec |
edge-rtac
An edge-rtac level request to get all open loans for greater than 300 holdings per instance
| GET /rtac for 30 minutes | Total Instances | Total Holdings | Average | 95th %tile | Average (Honeysuckle) | Delta | ||||||||
| 1 user1 vUser | 1 | 333 | 800 ms | 972 ms | 721 ms | 1 user+79 ms | ||||||||
| 1 vUser | 50 | mix of 3,4,5,6,7 | 807 ms | 1349 ms | 2.58 seconds | -1 | user.7 sec | |||||||
| 1 vUser | 50 | mix of 11,12 | 5796 ms | 10997 ms | - | - | ||||||||
| 2 users2 vUsers | 50 | mix of 11,12 | 6904 ms | 14619 ms | - | - | ||||||||
| 20 users20 vUsers | 50 | mix of 11,12 | 34725 ms | 57423 ms | - | - | ||||||||
| 1 user1 vUser | 30 | >=10 (one has 5k) from
| 4887 ms | 5165 ms | - | - | ||||||||
| 50 users50 vUsers | 30 | >=10 (one has 5k) from
| 63966 ms | 10123 ms | - | - | ||||||||
| 1 user1 vUser | 1 | >=10 (one has 5k) from
| 239 ms | 412 ms | - | - |
...
Average Time, ms | Module | Query | |
|---|---|---|---|
| 1 user1 vUser | 119 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
50 instances with a mix of 3,4,5,6,7 holdings per instance
Average Time, ms | Module | Query | |
|---|---|---|---|
| 20 users20 vUsers | 1057 ms | mod_inventory_storage | select * from get_items_and_holdings_view($1,$2) |
RDS CPU Utilization
1 user 1 instance with 333 holdings. CPU is around 24% which is normal.
.
1 user 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is around 27% which is normal.
.
15 users, 20 users 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU is close to 100% which is high because DB was fired with 15, 20 different users. This run involved 504 errors from mod-rtac. These many numbers of concurrent users are rare and will not happen in production.
Service Memory Utilization
15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. Service memory Utilization is relatively high for mod-rtac compared to other modules in the graph but this is rare. This use case was run to recreate the issue reported in MODRTAC-53. 2 vertical lines represent the time range for the test run.
.
Service CPU Utilization
15 users and 20 users for 50 instances with a mix of 3,4,5,6,7 holdings per instance. CPU Utilization is relatively high but this is rare. This use case was run to recreate the issue reported in MODRTAC-53
.
Appendix
Backend:
- mod-circulation-19.2.3
- mod-circulation-storage-12.1.2
- mod-inventory-storage-19.4.3
- mod-inventory-16.1.0
- okapi-4.2.2
- mod-authtoken-2.6.0
- mod-permissions-5.12.2
Environment:
...
99.5% for /inventory-hierarchy. For 1 user was about 18%
Service Memory Utilization
Service CPU Utilization
Appendix
- 14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 1 instance of db.r6.xlarge database instance: Writer instance
- MSK tenant [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
| qcp1-pvt | |||||||||
| mod-rtac | 5 | | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| mod-authtoken | 6 | | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
| mod-permissions | 10 | | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 |
| mod-inventory-storage | 5 | | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
| mod-circulation-storage | 5 | | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| mod-inventory | 5 | | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
| mod-circulation | 5 | | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| edge-rtac | 5 | | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| nginx-okapi | 5 | | 2 | 1024 | 896 | 128 | 0 | 0 | 0 |
| okapi | 5 | | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
Methodology/Approach
Run the test from the script (located in the "workflows-scripts/RTAC/rtac-all-for-regress" folder on GitHub) one by one. 8 tests as described in Test Runs.
There are pictures of Explain Analyze of long-running query:
