Overview

This is a report for a series of Check-in-check-out test runs against the Lotus release. 

Back End:

• mod-circulation-23.0.1

• mod-circulation-storage-14.0.0

• mod-inventory-18.1.4

• mod-inventory-storage-23.0.2

• mod-authtoken-2.9.1

• mod-pubsub-2.5.0

• mod-patron-blocks-1.5.0

• mod-feesfines-17.1.0

• okapi-4.13.0

Front End:

• Item Check-in (folio_checkin-7.0.0)

• Item Check-out (folio_checkout-8.0.0)

Infrastructure:

• 75 back-end modules deployed in 159 ECS tasks

• 3 okapi ECS services

• 9 m5.xlarge  EC2 instances

• 2 db.r6g.xlarge AWS RDS instance

• INFO logging level

High Level Summary

• Lotus performs much better than Kiwi response times for check-ins are mid 600ms, checkout 900ms - 1000ms, with little variations between 1 and 20 users

• Database performance is better and uses less CPU compared to Kiwi.

• Worst-performing APIs are only POST /checkin-by-barcode and POST /checkout-by-barcode.  They have improved since Kiwi, now averaging in the mid 300ms even in the 20 users scenario.

  • GET /circulation/loans takes 185ms in the 20 users case.

Test Runs

Results

Response Times (Average of all tests listed above, in seconds)

Response times are consistently good, sub-second for check-ins and a tad over 1 second for check-outs even in the 75th percentile. The variations between 1 and 20 users continue to get smaller (compared to previous releases)

Comparisons to Last Release

The following tables compare Kiwi test results against Lotus General release (deployed at Lotus Bugfest).

Response Time Comparison

In the tables below, the Delta columns express the differences between Kiwi and Lotus releases in percentage.

In general there are no regressions in performance but instead there are improvements in response times across the board.  Check in times improved by at least 15% and check out times consistently around 30%.  The 20 users' response times are also in very similar to 8 users, especially for checkouts. This means that the system is very stable from 1 to 20 users.

Note: LT = Lotus build, KW = Kiwi build

"Worst" API Comparisons

The APIs identified in the table below were the ones that took over 100ms to execute in Lotus and Kiwi in the 75th percentile.  In Lotus these APIs are generally better especially with 5 or more concurrent users. Impressively the response times of POST checkout-by-barcode and POST check-in-by-barcode all are under 400ms! GET inventory/items' response times are well under 100ms now compared to being on the border in the Kiwi release.

Average Response Time in milliseconds. 

Longevity Test

Longevity test shows that Check Out response time increased as time went on, but not as aggressive as in Kiwi.