Check-in-check-out Test Report (Morning Glory)
Overview
This is a report for a series of Check-in-check-out test runs against the Lotus release.
Back End:
- mod-circulation-23.0.11
- mod-circulation-storage-14.1.0
- mod-inventory-18.2.0
- mod-inventory-storage-24.0.0
- mod-authtoken-2.11.0
- mod-pubsub-2.6.0
- mod-patron-blocks-1.6.0
- mod-feesfines-18.0.0
- okapi-4.14.2
Front End:
- Item Check-in (folio_checkin-7.1.0)
- Item Check-out (folio_checkout-8.1.0)
Infrastructure:
- 79 back-end modules deployed in 153 ECS tasks (changed. In Lotus, it was75 back-end modules deployed in 159 ECS tasks)
- 3 okapi ECS tasks
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was 9 m5.xlarge)
- 2 db.r6g.xlarge AWS RDS instance
- INFO logging level
- mod-inventory (running tasks -2)
- 1024 CPU units, 1684 MiB
- mod-inventory-storage (running tasks -2)
- 1024 CPU units, 1684MB mem
- mod-circulation (running tasks -2)
- 1024 CPU units, 896MB mem
- mod-circulation-storage (running tasks -2)
- 1024 CPU units, 1024MB mem
- mod-pubsub (running tasks -2)
- 1024 CPU units, 1296 MiB
- mod-patron-blocks (running tasks -2)
- 1024 CPU units, 896MB mem
- mod-feesfines (running tasks -2)
- 128 CPU units, 896MB mem
- mod-authtoken (running tasks -2)
- 128 CPU units, 896MB mem
- okapi (running tasks -3)
- 1024 CPU units, 1360 MB mem
High-Level Summary
- Morning Glory performs much better than Lotus response times for check-ins are mid 400ms, checkout 600ms - 800ms, with little variations between 1 and 50 users
- Worst-performing APIs are only POST /checkin-by-barcode and POST /checkout-by-barcode. They have improved since Lotus, now averaging in the mid 200ms even in the 50 users scenario.
- GET /circulation/loans take 107ms on average in the 20 users and 50 users cases.
Test Runs
Test | Virtual Users | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 1 user | 30 mins | t3.medium | 3 |
2. | 5 users | 30 mins | t3.medium | 3 |
3. | 8 users | 30 mins | t3.medium | 3 |
4. | 20 users | 30 mins | t3.medium | 4 |
5. | 50 users | 30 mins | t3.large | 6 |
9. | 20 users longevity | 16 hours | t3.xlarge | 14 |
Results
Response Times (Average of all tests listed above, in seconds)
Average (seconds) | 50th %tile (seconds) | 75th %tile (seconds) | 95th %tile (seconds) | |||||
Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | |
1 user | 0.516 | 0.795 | 0.49 | 0.758 | 0.518 | 0.839 | 0.629 | 1.012 |
5 users | 0.418 | 0.646 | 0.408 | 0.624 | 0.433 | 0.672 | 0.493 | 0.766 |
8 users | 0.411 | 0.627 | 0.4 | 0.606 | 0.424 | 0.646 | 0.478 | 0.741 |
20 users | 0.401 | 0.615 | 0.393 | 0.6 | 0.413 | 0.632 | 0.456 | 0.704 |
50 users | 0.371 | 0.601 | 0.373 | 0.59 | 0.395 | 0.616 | 0.436 | 0.682 |
Response times are consistently good, and over 1 second only for check-out in the 95th percentile. The 50 users test has the best response times.
Comparisons to Last Release
The following tables compare Lotus test results against Morning Glory.
Response Time Comparison
In the tables below, the Delta columns express the differences between Lotus and Morning Glory releases in percentage.
In general, there are no regressions in performance instead, there are improvements in response times across the board. Check-in times improved by at least 30% and check-out times consistently around 35%. The 50 users' response times are also very similar to the 5 users. This means that the system is very stable from 1 to 50 users.
Note: LT = Lotus build, MG = Morning Glory build
Average | 50th percentile | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Check-in MG | Check-in LT | Delta | Check-out MG | Check-out LT | Delta | Check-in MG | Check-in LT | Delta | Check-out MG | Check-out LT | Delta | |
1 user | 0.516 | 0.73 | 29.32% | 0.795 | 1.188 | 33.08% | 0.49 | 0.682 | 28.15% | 0.758 | 1.084 | 30.07% |
5 users | 0.418 | 0.62 | 32.58% | 0.646 | 0.976 | 33.81% | 0.408 | 0.582 | 29.90% | 0.624 | 0.915 | 31.80% |
8 users | 0.411 | 0.615 | 33.17% | 0.627 | 0.971 | 35.43% | 0.4 | 0.581 | 31.15% | 0.606 | 0.904 | 32.96% |
20 users | 0.401 | 0.661 | 39.33% | 0.615 | 1.052 | 41.54% | 0.393 | 0.62 | 36.61% | 0.6 | 0.967 | 37.95% |
50 users | 0.371 | - | - | 0.601 | - | - | 0.373 | - | - | 0.59 | - | - |
75th percentile | 95th percentile | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Check-in MG | Check-in LT |