Check-in-check-out Test Report (Kiwi)
Overview
This is a report for a series of Check-in-check-out test runs against the Kiwi release.
Back End:
- mod-circulation-22.1.0
- mod-circulation-storage-13.1.0
- mod-inventory-18.0.0
- mod-inventory-storage-22.0.0
- mod-authtoken-2.9.0
- mod-pubsub-2.4.0
- mod-patron-blocks-1.4.0
- mod-feesfines-17.0.0
- okapi-4.9.0
Front End:
- folio_circulation-6.0.0
- Item Check-in (folio_checkin-6.0.0)
- Item Check-out (folio_checkout-7.0.0)
Infrastructure:
- 71 back-end modules deployed in 141 ECS tasks
- 3 okapi ECS services
- 6 m5.xlarge EC2 instances
- 2 db.r6g.xlarge AWS RDS instance
- INFO logging level
High Level Summary
- Kiwi performs much better than Juniper GA, response times for check-ins are <900ms, checkout <1500ms, with not much variation between 1 and 20 users
- Database performance is better and uses much less CPU compared to Juniper GA
- Worst-performing APIs are still POST /checkin-by-barcode and POST /checkout-by-barcode. The response time are still about 500ms. GET /circulation/loans also takes more than 200ms. GET /inventory/item (by barcode) takes less than 100ms now.
- Longevity test shows response times worsen over time, probably due to the growing DB CPU utilization. - CIRCSTORE-304Getting issue details... STATUS Potentially could address this situation
Test Runs
Test | Virtual Users | Duration |
1. | 1 | 30 mins |
2. | 5 | 30 mins |
3. | 8 | 30 mins |
4. | 20 | 30 mins |
5. | 1 (repeat) | 30 mins |
Results
Response Times (Average of all tests listed above, in seconds)
Average (seconds) | 50th %tile (seconds) | 75th %tile (seconds) | 95th %tile (seconds) | |||||
Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | |
1 user | 0.838 | 1.582 | 0.767 | 1.464 | 0.915 | 1.716 | 0.915 | 1.716 |
5 users | 0.73 | 1.376 | 0.676 | 1.272 | 0.769 | 1.448 | 1.052 | 2.061 |
8 users | 0.758 | 1.392 | 0.674 | 1.228 | 0.839 | 1.44 | 1.202 | 2.343 |
20 users | 0.899 | 1.506 | 0.731 | 1.303 | 0.931 | 1.535 | 1.531 | 2.504 |
Response times are consistently good, sub-second for check-ins and around 1.5 seconds for check-outs even in the 75th percentile. Not much variations between 1 and 20 users.
Comparisons to Last Release
The following tables compare Juniper HF3 test results against Kiwi General release (deployed at Bugfest).
Response Time Comparison
In general there is no regression in performance. The response times between Kiwi and Juniper are very close to each other for 1-8 users load unless they were in the 95th percentile group or the 20 users load where Kiwi clearly out-perform Juniper. In the tables below, the Delta columns express the differences between Juniper and Kiwi releases in percentage. Any percentage +/-5% is not statistically is within the margin of error. It is also noteworthy that Kiwi seems to invoke the GET /automated-patron-blocks 3 times instead of once - UICHKOUT-755Getting issue details... STATUS . This call averages 25ms under all loads, so if 2 of these 3 calls were not needed (why would the UI calls it thrice?) then the Kiwi's Checkout average response times could improve by another 50ms.
Note: JP = Juniper build, KW = Kiwi build
Average | 50th percentile | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Check-in JP | Check-in KW | Delta | Check-out JP | Check-out KW | Delta | Check-in JP | Check-in KW | Delta | Check-out JP | Check-out KW | Delta | |
1 user | 0.944 | 0.838 | 11.23% | 1.579 | 1.582 | -0.19% | 0.835 | 0.767 | 8.14% | 1.411 | 1.464 | -3.76% |
5 users | 0.811 | 0.73 | 9.99% | 1.359 | 1.376 | -1.25% | 0.750 | 0.676 | 9.87% | 1.23 | 1.272 | -3.41% |
8 users | 0.889 | 0.758 | 14.74% | 1.425 | 1.392 | 2.32% | 0.785 | 0.674 | 14.14% | 1.262 | 1.228 | 2.69% |
20 users | 1.386 | 0.899 | 35.14% | 2.21 | 1.506 | 31.86% | 1.172 | 0.731 | 37.63% | 1.887 | 1.303 | 30.95% |