WIP
Work in progress...
Overview
In this workflow, we are checking the performance of POST /loan-storage/loans API for creating loans in the Honeysuckle release - PERF-148 - Getting issue details... STATUS IC's normally create loans in bulk using mod-circulation business logic which is slow. This is an investigation if the same can be done using the storage module bypassing business logic.
We tested with 1 virtual user for 30 minutes.
Backend:
- mod-circulation-storage-12.1.4
- okapi-4.3.3
- mod-authtoken-2.6.0
Environment:
- 7.2 million UChi SRS records
- 7.2 million inventory records
- 63 FOLIO back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 10 m5.large EC2 instances
- 1 writer db.r5.xlarge 1 reader db.r5.xlarge AWS RDS instance
- INFO logging level
High-Level Summary
- As the number of instances to export increases from 1 - 500,000 for 1, 2 users, performance improves. For all cases, mod-data-export, mod-source-record-storage is stable and performant.
- data-export/job-executions API is polled every 3 seconds vs 5 seconds in Goldenrod. This means there are more requests overall but it also improves data-export execution. Data-export job completes faster(around 2 seconds) compared to the previous release. Other than that, data-export is stable and see performance improvements compared to the Goldenrod release.
- A new feature added for data-export is it will kill failed job every 6 hours
Test Runs
1 user can create 71099 loans in 30 minutes by making a request to POST /loan-storage/loans
| Requests | Duration | Total loan requests | Success | Fail | Req/s | Min | 75th pct | Max(millisecond) | Average (millisecond) | Latency (milliseconds) |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-circulation-storage POST /loan-storage/loans | 30 min | 71099 | 71099 | 0 | 39.499 | 0.018 | 25 | 34.9 | 25 | 37 |
SQL query involved during POST /loan-storage/loans
| SQL query | Response time (Avg) |
|---|---|
| INSERT INTO fs09000000_mod_circulation_storage.loan (id, jsonb) VALUES ($1, $2) RETURNING jsonb | 1 millisecond |
CPU Utilization
CPU utilization increases significantly for okapi but for other modules, it is relatively normal.
Service Memory Utilization
Memory is stable for all modules involved. Okapi is relatively higher but stable at 90%
