IN PROGRESS
PERF-39 - Getting issue details... STATUS
Overview
In this workflow, we are checking the performance of the renewal of loan items for the Goldenrod release.
We tested it with 1, 2 virtual users for 30 minutes.
Backend:
- mod-users-17.1.0
- mod-users-bl-6.0.0
- mod-authtoken-2.5.1
- mod-permissions-5.11.2
- mod-circulation-19.0.9
- mod-circulation-storage-12.0.1
- okapi-3.1.1
Environment:
- 55 back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 8 m5.large EC2 instances
- 2 (1 reader, 1 writer) db.r5.xlarge AWS RDS instance
- INFO logging level
High Level Summary
- On average POST_/circulation/renew-by-barcode API call takes average 671.12 seconds as we gradually increase the number of loans and users to 200 and 2 respectively.
- POST_/circulation/renew-by-barcode API makes 26 different API requests to gather userBarcode and itemBarcode information. 34% of those are calls are to mod-authtoken. See Recommended Improvements for the JIRAs that were covered by this testing effort.
- No missing indexes or no suspicious slow queries found except a number of SELECT count_estimate() calls.
Test Runs
Test | Virtual Users | Duration | OKAPI log level | Loans per User |
1. | 1 | 30 min | WARNING | 10, 50, 100, 200 |
2. | 2 | 30 min | WARNING | 10, 50, 100, 200 |
| 3. | 1 | 30 min | INFO | 200 |
Results
Renew all loans by barcode
Total Users for 30 minutes Loans Renew all loans (Average) POST_/circulation/renew-by-barcode (Average) 1 user 10 6.52 seconds
585.11 ms
50 31.81 seconds
637.48 ms
100 1.06 minutes 650.15 ms
200 2.09 minutes 673.40 ms
2 users 10 7.42 seconds
683.76 ms
50 32.61 seconds
686.65 ms
100 1.10 minutes
705.37 ms
200 2.27 minutes
747.04 ms
- Below is the dependency graph generated from Giraffe for single POST_/circulation/renew-by-barcode API. It shows individual calls made by renew-by-barcode to gather prerequisite data from different APIs such as item-storage, location-units, loan-types, request-storage, notice-policy, locations, and mod-authtoken. In all, this call took 990 ms, and 6% of that is consumed by mod-authtoken and the remaining 94% of the time is taken by other APIs to gather prerequisite data.
Database CPU Utilization
The database CPU usage increases gradually for 1, 2 users run. At maximum 9% CPU usage for 1 user. For 2 users, max CPU utilization increases and then stabilizes as the number of loans increases, mostly because RDS Postgresql is caching the most frequent query results. Therefore, CPU utilization is relatively on the lower side.
| For 30 minutes | Loans | RDS CPU Utilization |
| 1 user | 10 | 7% |
| 50 | 8% | |
| 100 | 8% | |
| 200 | 9% | |
| 2 users | 10 | 14% |
| 50 | 15% | |
| 100 | 15% | |
| 200 | 15% |
Database queries
For 1 run of 30 minutes test, following queries were made. To point out, total calls made and the total time spend on these queries is significant.
| Query | Total time | Average Time | Total calls made |
|---|---|---|---|
| SELECT count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.item WHERE lower(f_unaccent(item.jsonb->>''barcode'')) LIKE lower(f_unaccent(''54357881''))') | 2 minutes or 71% | 14 ms | 9027 |
| SELECT count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>''userId'') = ''4deba809-4fb4-432a-b3df-c54b6ad51d22''') | 2 ms or 10% | 2 ms | 8457 |
Service CPU Utilization
Service CPU utilization is relatively on a lower side for most modules except for mod-circulation
| For 30 minutes | Modules | Service CPU Utilization |
| 1 user | mod-inventory | 3% |
| mod-inventory-storage | 10% | |
| mod-circulation | 80% | |
| mod-circulation-storage | 6% | |
| 2 users | mod-inventory | 3.25% |
| mod-inventory-storage | 17.75% | |
| mod-circulation | 81.75% | |
| mod-circulation-storage | 11.75% |
Service Memory Utilization
Service memory was stable for most modules except mod-circulation which was on the higher side.
| For 30 minutes | Modules | Memory CPU Utilization |
| 1 user | mod-inventory | 63% |
| mod-inventory-storage | 56% | |
| mod-circulation | 115% | |
| mod-circulation-storage | 42% | |
| 2 users | mod-inventory | 65% |
| mod-inventory-storage | 56% | |
| mod-circulation | 115% | |
| mod-circulation-storage | 49% |
Recommended Improvements
- The following JIRAs has been created for mod-authtoken optimization:
MODAT-80 - Getting issue details... STATUS
MODAT-83 - Getting issue details... STATUS
Appendix
See Attached user_loanRenewals-test-report.xlsx for details