IN PROGRESS
PERF-39 - Getting issue details... STATUS
Overview
In this workflow, we are checking the performance of the renewal of loan items for the Goldenrod release.
We tested it with 1, 2 virtual users for 30 minutes.
Backend:
- mod-users-17.1.0
- mod-users-bl-6.0.0
- mod-authtoken-2.5.1
- mod-permissions-5.11.2
- mod-circulation-19.0.9
- mod-circulation-storage-12.0.1
- okapi-3.1.1
Environment:
- 55 back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 8 m5.large EC2 instances
- 2 (1 reader, 1 writer) db.r5.xlarge AWS RDS instance
- INFO logging level
High Level Summary
- On average POST_/circulation/renew-by-barcode API call takes average 671.12 seconds as we gradually increase the number of loans and users to 200 and 2 respectively.
- POST_/circulation/renew-by-barcode API makes 26 different API requests to gather userBarcode and itemBarcode information. 34% of those are calls are to mod-authtoken. See Recommended Improvements for the JIRAs that were covered by this testing effort.
- No missing indexes or no suspicious slow queries found except a number of SELECT count_estimate() calls.
Test Runs
Test | Virtual Users | Duration | OKAPI log level | Loans per User |
1. | 1 | 30 min | WARNING | 10, 50, 100, 200 |
2. | 2 | 30 min | WARNING | 10, 50, 100, 200 |
| 3. | 1 | 30 min | INFO | 200 |
Results
Renew all loans by barcode
Total Users for 30 minutes Loans Renew all loans (Average) POST_/circulation/renew-by-barcode (Average) 1 user 10 6.52 seconds
585.11 ms
50 31.81 seconds
637.48 ms
100 1.06 minutes 650.15 ms
200 2.09 minutes 673.40 ms
2 users 10 7.42 seconds
683.76 ms
50 32.61 seconds
686.65 ms
100 1.10 minutes
705.37 ms
200 2.27 minutes
747.04 ms
- Below is the dependency graph generated from Giraffe for single POST_/circulation/renew-by-barcode API. It shows individual calls made by renew-by-barcode to gather prerequisite data from different APIs such as item-storage, location-units, loan-types, request-storage, notice-policy, locations, and mod-authtoken. In all, this call took 990 ms, and 6% of that is consumed by mod-authtoken and the remaining 94% of the time is taken by other APIs to gather prerequisite data.
Database CPU Utilization
The database CPU usage increases gradually for 1, 2 users run. At maximum 9% CPU usage for 1 user.
For 2 users, max CPU utilization increases and then stabilizes as the number of loans increases. Therefore, CPU utilization is relatively on the lower side.
| For 30 minutes | Loans | RDS CPU Utilization |
| 1 user | 10 | 7% |
| 50 | 8% | |
| 100 | 8% | |
| 200 | 9% | |
| 2 users | 10 | 14% |
| 50 | 15% | |
| 100 | 15% | |
| 200 | 15% |
Database queries
For 1 run of 30 minutes test, following queries were made. To point out, total calls made and the total time spend on these queries is significant.
| Query | Total time | Average Time | Total calls made |
|---|---|---|---|
| SELECT count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.item WHERE lower(f_unaccent(item.jsonb->>''barcode'')) LIKE lower(f_unaccent(''54357881''))') | 71% | 14 ms | 9027 |
| SELECT count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>''userId'') = ''4deba809-4fb4-432a-b3df-c54b6ad51d22''') | 10% | 2 ms | 8457 |
Service CPU Utilization
Service CPU utilization is relatively on a lower side for most modules except for mod-circulation
| For 30 minutes | Modules | Service CPU Utilization |
| 1 user | mod-inventory | 3% |
| mod-inventory-storage | 10% | |
| mod-circulation | 95% | |
| mod-circulation-storage | 6% | |
| 2 users | mod-inventory | 3.25% |
| mod-inventory-storage | 17.75% | |
| mod-circulation | 122.75% | |
| mod-circulation-storage | 11.75% |
Service Memory Utilization
Service memory was stable for most modules except mod-circulation which was on the higher side.
| For 30 minutes | Modules | Memory CPU Utilization |
| 1 user | mod-inventory | 63% |
| mod-inventory-storage | 56% | |
| mod-circulation | 118% | |
| mod-circulation-storage | 42% | |
| 2 users | mod-inventory | 65% |
| mod-inventory-storage | 56% | |
| mod-circulation | 118% | |
| mod-circulation-storage | 49% |
Graph for service memory utilization for mod-circulation. There seems to be a potential memory leak problem. Memory usage started at ~112%, rose to ~118% and fall back to ~101%
Recommended Improvements
- The following JIRAs has been created for mod-authtoken optimization:
MODAT-80 - Getting issue details... STATUS
MODAT-83 - Getting issue details... STATUS
Appendix
See Attached user_loanRenewals-test-report.xlsx for details