Overview

This is a report for a series of Check-in-check-out test runs against the Lotus release.

Back End:

mod-circulation-23.0.11
mod-circulation-storage-14.1.0
mod-inventory-18.2.0
mod-inventory-storage-24.0.0
mod-authtoken-2.11.0
mod-pubsub-2.6.0
mod-patron-blocks-1.6.0
mod-feesfines-18.0.0
okapi-4.14.2

Front End:

Item Check-in (folio_checkin-7.1.0)
Item Check-out (folio_checkout-8.1.0)

Infrastructure:

79 back-end modules deployed in 153 ECS tasks (changed. In Lotus, it was75 back-end modules deployed in 159 ECS tasks)
3 okapi ECS tasks
10 m6i.2xlarge EC2 instances (changed. In Lotus it was 9 m5.xlarge)
2 db.r6g.xlarge AWS RDS instance
INFO logging level
mod-inventory (running tasks -2)
- 1024 CPU units, 1684 MiB
mod-inventory-storage (running tasks -2)
- 1024 CPU units, 1684MB mem
mod-circulation (running tasks -2)
- 1024 CPU units, 896MB mem
mod-circulation-storage (running tasks -2)
- 1024 CPU units, 1024MB mem
mod-pubsub (running tasks -2)
- 1024 CPU units, 1296 MiB
mod-patron-blocks (running tasks -2)
- 1024 CPU units, 896MB mem
mod-feesfines (running tasks -2)
- 128 CPU units, 896MB mem
mod-authtoken (running tasks -2)
- 128 CPU units, 896MB mem
okapi (running tasks -3)
- 1024 CPU units, 1360 MB mem

High-Level Summary

Morning Glory performs much better than Lotus response times for check-ins are mid 400ms, checkout 600ms - 800ms, with little variations between 1 and 50 users
Worst-performing APIs are only POST /checkin-by-barcode and POST /checkout-by-barcode. They have improved since Lotus, now averaging in the mid 200ms even in the 50 users scenario.
- GET /circulation/loans take 107ms on average in the 20 users and 50 users cases.

Test Runs

Test	Virtual Users	Duration	Load generator size (recommended)	Load generator Memory(GiB) (recommended)
1.	1 user	30 mins	t3.medium	3
2.	5 users	30 mins	t3.medium	3
3.	8 users	30 mins	t3.medium	3
4.	20 users	30 mins	t3.medium	4
5.	50 users	30 mins	t3.large	6
9.	20 users longevity	16 hours	t3.xlarge	14

Results

Response Times (Average of all tests listed above, in seconds)

	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)
	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out
1 user	0.516	0.795	0.49	0.758	0.518	0.839	0.629	1.012
5 users	0.418	0.646	0.408	0.624	0.433	0.672	0.493	0.766
8 users	0.411	0.627	0.4	0.606	0.424	0.646	0.478	0.741
20 users	0.401	0.615	0.393	0.6	0.413	0.632	0.456	0.704
50 users	0.371	0.601	0.373	0.59	0.395	0.616	0.436	0.682

Response times are consistently good, and over 1 second only for check-out in the 95th percentile. The 50 users test has the best response times.

Comparisons to Last Release

The following tables compare Lotus test results against Morning Glory.

Response Time Comparison

In the tables below, the Delta columns express the differences between Lotus and Morning Glory releases in percentage.

In general, there are no regressions in performance instead, there are improvements in response times across the board. Check-in times improved by at least 30% and check-out times consistently around 35%. The 50 users' response times are also very similar to the 5 users. This means that the system is very stable from 1 to 50 users.

Note: LT = Lotus build, MG = Morning Glory build

	Average						50th percentile
	Check-in MG	Check-in LT	Delta	Check-out MG	Check-out LT	Delta	Check-in MG	Check-in LT	Delta	Check-out MG	Check-out LT	Delta
1 user	0.516	0.73	29.32%	0.795	1.188	33.08%	0.49	0.682	28.15%	0.758	1.084	30.07%
5 users	0.418	0.62	32.58%	0.646	0.976	33.81%	0.408	0.582	29.90%	0.624	0.915	31.80%
8 users	0.411	0.615	33.17%	0.627	0.971	35.43%	0.4	0.581	31.15%	0.606	0.904	32.96%
20 users	0.401	0.661	39.33%	0.615	1.052	41.54%	0.393	0.62	36.61%	0.6	0.967	37.95%
50 users	0.371	-	-	0.601	-	-	0.373	-	-	0.59	-	-

	75th percentile						95th percentile
	Check-in MG	Check-in LT	Delta	Check-out MG	Check-out LT	Delta	Check-in MG	Check-in LT	Delta	Check-out MG	Check-in LT	Delta
1 user	0.518	0.774	33.07%	0.839	1.24	32.34%	0.629	1.000	37.10%	1.012	1.812	44.15%
5 users	0.433	0.662	34.59%	0.672	1.02	34.12%	0.493	0.844	41.59%	0.766	1.440	46.8%
8 users	0.424	0.654	35.17%	0.646	1.006	35.79%	0.478	0.836	42.82%	0.741	1.428	48.1%
20 users	0.413	0.702	41.17%	0.632	1.107	42.91%	0.456	0.918	50.32%	0.704	1.593	55.8%
50 users	0.395	-	-	0.616	-	-	0.436	-	-	0.682	-	-

"Worst" API Comparisons

The APIs identified in the table below were the ones that took over 100ms to execute in Lotus and Kiwi in the 75th percentile. In Morning Glory these APIs are generally better, especially with 5 or more concurrent users. Impressively the response times of POST-checkout-by-barcode and POST-check-in-by-barcode all are under 300ms! GET inventory/items' response times are well under 60ms now compared to being on the border in the Lotus release.

Average Response Time in milliseconds.

API	1 user MG (75th %tile)	1 user LT (75th %tile)	5 users MG (75th %tile)	5 users LT (75th %tile)	8 users MG (75th %tile)	8 users LT (75th %tile)	20 users MG (75th %tile)	20 users LT (75th %tile)	50 users MG (75th %tile)	50 users LT (75th %tile)
POST checkout-by-barcode	296	435	217	330	211	349	204	384	205	-
POST checkin-by-barcode	275	386	199	298	199	315	189	348	186	-
GET circulation/loans	152	222	113	161	109	172	107	196	107	-
GET inventory/items	60	97	44	71	40	67	39	75	41	-

Longevity Test

Longevity test shows that Check Out response time increased as time went on, but not as aggressive as in Lotus.

	Check In	Check Out
1st Hour	0.389s	0.633s
8th Hour	0.348s	0.666s
15th Hour	0.364s	0.754s

In the response time graph below the Checkout Controller time, which gathers all check-out API response times), increased over the 16-hours window, from just 0.633s to 0.754s.

The DB CPU utilization percentage did not increase over time and was about10% by the end of the test. There are huge spikes every 30 minutes. These are due to the background tasks that run periodically and as the number of loans grew so did the physical resources needed to process all of them.

The number of connections also rose over time, from 348 to 391 connections. It's unclear what (DB connections) caused the DB to use more CPU resources as the test progressed.

Database's memory dipped a bit but no symptoms of memory leaks. The memory level bounced back up right after the test finished.

Modules CPU Utilization During Longevity Test

Here is a view of the CPU utilization. A couple of observations:

mod-authtoken seems to take up CPU resources in a cyclical way from 28% to 32%.
Okapi uses only about 12%CPU on average compared to Lotus about 450-470% CPU on average.
mod-users CPU Utilization grows from 20% up to 36%
Other modules used less than 20% CPU on average.
mod-circulation-storage spikes periodically due to processing background tasks

Here is the Service CPU Utilization graph with the main involved mods.

There do not appear to be any memory leak issues in Morning Glory. There were no spikes and the processes did not crash.

The timeline highlighted below encompasses all 5 test runs (1, 5, 8, 20, and 50 users).

Modules CPUs and Memory Utilization

The relevant services overall seem to occupy CPU resources nominally. Only mod-authtoken seems to have the spikes but the processes did not crash.

20-users tests	Avg	Max
mod-users	18%	18%
mod-pubsub	5%	5%
okapi	11%	11%
mod-circulation	4%	4%
mod-circulation-storage	5%	5%
mod-inventory	6%	6%
mod-inventory-storage	7%	7%
mod-patron-blocks	1%	1%
mod-feesfines	14%	15%
mod-authtoken	24%	66%

50-users tests	Avg	Max
mod-users	38%	39%
mod-pubsub	7%	7%
okapi	25%	26%
mod-circulation	6%	7%
mod-circulation-storage	9%	10%
mod-inventory	7%	7%
mod-inventory-storage	12%	14%
mod-patron-blocks	2%	2%
mod-feesfines	31%	33%
mod-authtoken	47%	116%

Services' memory seems to be stable during the test runs.

	Avg	Max
mod-users	36%	36%
mod-pubsub	45%	45%
okapi	50%	50%
mod-circulation	80%	80%
mod-circulation-storage	55%	56%
mod-inventory	91%	91%
mod-inventory-storage	71%	74%
mod-patron-blocks	44%	44%
mod-feesfines	39%	39%
mod-authtoken	30%	30%

Database and network

The freeable memory metric shows how much memory is being consumed and remaining during a test run. In the graph below memory usage is pretty good and stable throughout the test runs but generally slightly trending down after each run and bounced back almost to the starting point after each test. This does not mean there is a memory leak because the 16 hours longevity test does not reveal any leaks (see above).