WORK IN PROGRESS
PERF-39 - Getting issue details... STATUS
Overview
In this workflow, we are checking the performance of the renewal of loan items for the Goldenrod release.
We tested it with 1, 2 virtual users for 30 minutes.
Backend:
- mod-users-17.1.0
- mod-users-bl-6.0.0
- mod-authtoken-2.5.1
- mod-permissions-5.11.2
- mod-circulation-19.0.9
- mod-circulation-storage-12.0.1
- okapi-3.1.1
Environment:
- 55 back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 8 m5.large EC2 instances
- 1 db.r5.xlarge AWS RDS instance
- INFO logging level
High Level Summary
- On average POST_/circulation/renew-by-barcode API call takes average 671.12 seconds as we gradually increase the number of loans and users to 200 and 2 respectively.
- POST_/circulation/renew-by-barcode API makes 26 different API requests to gather userBarcode and itemBarcode information. 34% of those are calls are to mod-authtoken. See Recommended Improvements for the JIRAs that were covered by this testing effort.
- No missing indexes or no suspicious slow queries found except a number of SELECT count_estimate() calls.
Test Runs
Test | Virtual Users | Duration | OKAPI log level | Profiled | Loans per User |
1. | 1 | 30 min | WARNING | No | 10, 50, 100, 200 |
2. | 2 | 30 min | WARNING | No | 10, 50, 100, 200 |
| 3. | 1 | 30 min | INFO | No | 10, 50, 100, 200 |
Results
Renew all loans by barcode
For 30 minutes Loans Renew all loans (Average) POST_/circulation/renew-by-barcode (Average) 1 user 10 6.52 seconds
585.11 ms
50 31.81 seconds
637.48 ms
100 1.06 minutes 650.15 ms
200 2.09 minutes 673.40 ms
2 users 10 7.42 seconds
683.76 ms
50 32.61 seconds
686.65 ms
100 1.10 minutes
705.37 ms
200 2.27 minutes
747.04 ms
- Below is the dependency graph generated from Giraffe for single POST_/circulation/renew-by-barcode API. It shows individual calls made by renew-by-barcode to gather prerequisite data from different APIs such as item-storage, location-units, loan-types, request-storage, notice-policy, locations, and mod-authtoken. In all, this call took 990 ms, and 6% of that is consumed by mod-authtoken and the remaining 94% of the time is taken by other APIs to gather prerequisite data.
Database
The database CPU usage decreases slightly as the number of users increases 1, 5 users run. At maximum 28% CPU usage for 1 user. We are seeing a slight decrease in CPU utilization for 5 Users because RDS Postgresql is caching the most frequent query results.
Database queries
For 30 minutes test, following queries were made
| Query | Total time | Average Time | Total calls made |
|---|---|---|---|
| SELECT count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.item WHERE lower(f_unaccent(item.jsonb->>''barcode'')) LIKE lower(f_unaccent(''54357881''))') | 2 minutes or 71% | 14 ms | 9027 |
| SELECT count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>''userId'') = ''4deba809-4fb4-432a-b3df-c54b6ad51d22''') | 0 minutes or 10% | 2 ms | 8457 |
CPU Utilization
CPU utilization is relatively on a lower side for most modules except for mod-inventory-storage
| 1 user | 5 users | |
|---|---|---|
| Average % | Average % | |
| mod-circulation | 3.14 | 3.95 |
| mod-circulation-storage | 4.84 | 6.38 |
| mod-inventory | 5.38 | 6.54 |
| mod-inventory-storage | 26 | 24 |
| mod-rtac | 2.45 | 3 |
Memory
Memory was stable throughout the runs, only a spike here or there, but in a 30 minutes run they were consistent.
| 1 user | 5 users | |
|---|---|---|
| Average % | Average % | |
| mod-circulation | 72.1 | 72.50 |
| mod-circulation-storage | 39.46 | 39.55 |
| mod-inventory | 44 | 46.5 |
| mod-inventory-storage | 44.30 | 45 |
| mod-rtac | 55.2 | 55 |
Recommended Improvements
- The following JIRA has been created for mod-authtoken optimization:
MODAT-80 - Getting issue details... STATUS
Appendix
See Attached rtac-test-report.xlsx for details