| Table of Contents | ||
|---|---|---|
|
Overview
This is a report for a series of Check-in-check-out test runs against the Lotus release.
Back End:
...
...
...
...
...
Front End:
- Item Check-in (folio_checkin-7.1.0)
- Item Check-out (folio_checkout-8.1.0)
Infrastructure:
...
- 1024 CPU units, 1684 MiB
...
- 1024 CPU units, 1684MB mem
...
- 1024 CPU units, 896MB mem
...
- 1024 CPU units, 1024MB mem
...
- 1024 CPU units, 1296 MiB
...
- 1024 CPU units, 896MB mem
...
- 128 CPU units, 896MB mem
...
- 1024 CPU units, 1360 MB mem
High Level Summary
- Morning Glory performs much better than Lotus response times for check-ins are mid 400ms, checkout 600ms - 800ms, with little variations between 1 and 50 users
- Worst-performing APIs are only POST /checkin-by-barcode and POST /checkout-by-barcode. They have improved since Lotus, now averaging in the mid 200ms even in the 50 users scenario.
- GET /circulation/loans takes 107ms in the 20 users and 50 users cases.
Test Runs
...
Test
...
Virtual Users
...
Duration
...
1.
...
2.
...
3.
...
4.
...
Results
Response Times (Average of all tests listed above, in seconds)
...
Response times are consistently good, and over 1 second only for check-out in the 95th percentile. The 50 users test has the best response times.
Comparisons to Last Release
The following tables compare Lotus test results against Morning Glory.
Response Time Comparison
...
In general, there are no regressions in performance but instead there are improvements in response times across the board. Check in times improved by at least 30% and check-out times consistently around 35%. The 50 users' response times are also in very similar to 5 users. This means that the system is very stable from 1 to 50 users.
Note: LT = Lotus build, MG = Kiwi build
...
| Table of Contents | ||
|---|---|---|
|
Overview
This is a report for a series of Check-in-check-out test runs against the Lotus release.
Back End:
- mod-circulation-23.0.11
- mod-circulation-storage-14.1.0
- mod-inventory-18.2.0
- mod-inventory-storage-24.0.0
- mod-authtoken-2.11.0
- mod-pubsub-2.6.0
- mod-patron-blocks-1.6.0
- mod-feesfines-18.0.0
- okapi-4.14.2
Front End:
- Item Check-in (folio_checkin-7.1.0)
- Item Check-out (folio_checkout-8.1.0)
Infrastructure:
- 79 back-end modules deployed in 153 ECS tasks (changed. In Lotus, it was75 back-end modules deployed in 159 ECS tasks)
- 3 okapi ECS tasks
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was 9 m5.xlarge)
- 2 db.r6g.xlarge AWS RDS instance
- INFO logging level
- mod-inventory (running tasks -2)
- 1024 CPU units, 1684 MiB
- mod-inventory-storage (running tasks -2)
- 1024 CPU units, 1684MB mem
- mod-circulation (running tasks -2)
- 1024 CPU units, 896MB mem
- mod-circulation-storage (running tasks -2)
- 1024 CPU units, 1024MB mem
- mod-pubsub (running tasks -2)
- 1024 CPU units, 1296 MiB
- mod-patron-blocks (running tasks -2)
- 1024 CPU units, 896MB mem
- mod-feesfines (running tasks -2)
- 128CPU units, 896MB mem
- mod-authtoken (running tasks -2)
- 128 CPU units, 896MB mem
- okapi (running tasks -3)
- 1024 CPU units, 1360 MB mem
High-Level Summary
- Morning Glory performs much better than Lotus response times for check-ins are mid 400ms, checkout 600ms - 800ms, with little variations between 1 and 50 users
- Worst-performing APIs are only POST /checkin-by-barcode and POST /checkout-by-barcode. They have improved since Lotus, now averaging in the mid 200ms even in the 50 users scenario.
- GET /circulation/loans take 107ms on average in the 20 users and 50 users cases.
Test Runs
Test | Virtual Users | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | 1 user | 30 mins | t3.medium | 3 |
2. | 5 users | 30 mins | t3.medium | 3 |
3. | 8 users | 30 mins | t3.medium | 3 |
4. | 20 users | 30 mins | t3.medium | 4 |
| 5. | 50 users | 30 mins | t3.large | 6 |
| 9. | 20 users longevity | 16 hours | t3.xlarge | 14 |
Results
Response Times (Average of all tests listed above, in seconds)
| Average (seconds) | 50th %tile (seconds) | 75th %tile (seconds) | 95th %tile (seconds) | |||||
| Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | Check-in | Check-out | |
| 1 user | 0.516 | 0.795 | 0.49 | 0.758 | 0.518 | 0.839 | 0.629 | 1.012 |
| 5 users | 0.418 | 0.646 | 0.408 | 0.624 | 0.433 | 0.672 | 0.493 | 0.766 |
| 8 users | 0.411 | 0.627 | 0.4 | 0.606 | 0.424 | 0.646 | 0.478 | 0.741 |
| 20 users | 0.401 | 0.615 | 0.393 | 0.6 | 0.413 | 0.632 | 0.456 | 0.704 |
| 50 users | 0.371 | 0.601 | 0.373 | 0.59 | 0.395 | 0.616 | 0.436 | 0.682 |
Response times are consistently good, and over 1 second only for check-out in the 95th percentile. The 50 users test has the best response times.
Comparisons to Last Release
The following tables compare Lotus test results against Morning Glory.
Response Time Comparison
In the tables below, the Delta columns express the differences betweenLotus and Morning Glory releases in percentage.
In general, there are no regressions in performance instead, there are improvements in response times across the board. Check-in times improved by at least 30% and check-out times consistently around 35%. The 50 users' response times are also very similar to the 5 users. This means that the system is very stable from 1 to 50 users.
Note: LT = Lotus build, MG = Kiwi build
| Average | 50th percentile | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Check-in MG | Check-in LT | Delta | Check-out MG | Check-out LT | Delta | 1 user | 0.516 | 0Check-in MG | Check-in LT | Delta | Check-out MG | Check-out LT | Delta |
| 1 user | 0.516 | 0.73 | 29.32% | 0.795 | 1.188 | 33.08% | 0.49 | 0.682 | 28.15% | 0.758 | 1.084 | 30.07% | |
| 5 users | 0.418 | 0.62 | 32.58% | 0.646 | 0.976 | 33.81% | 0.408 | 0.582 | 29.90% | 0.624 | 0.915 | 31.80% | |
| 8 users | 0.411 | 0.615 | 33.17% | 0.627 | 0.971 | 35.43% | 0.4 | 0.581 | 31.15% | 0.606 | 0.904 | 32.96% | |
| 20 users | 0.401 | 0.661 | 39.33% | 0.615 | 1.052 | 41.54% | 0.393 | 0.62 | 36.61% | 0.6 | 0.967 | 37.95% | |
| 50 users | 0.371 | - | - | 0.601 | - | - | 0.373 | - | - | 0.59 | - | - | |
...
The APIs identified in the table below were the ones that took over 100ms to execute in Lotus and Kiwi in the 75th percentile. In Morning Glory these APIs are generally better, especially with 5 or more concurrent users. Impressively the response times of POST-checkout-by-barcode and POST-check-in-by-barcode all are under 300ms! GET inventory/items' response times are well under 60ms now compared to being on the border in the Lotus release.
...
Longevity test shows that Check Out response time increased as time went on, but not as aggressive as in Lotus.
...
In the response time graph below the Checkout Controller time, which gathers all check-out API response times), increased over the 16-hours window, from just 0.633s to 0.754s.
The DB CPU utilization percentage did not increase over time, and was about10% by the end of the test. There are huge spikes every 30 minutes. These are due to the background tasks that run periodically and as the number of loans grew so did the physical resources needed to process all of them.
The number of connections also rose over time, from 348 to 391 connections. It's unclear what (DB connections) caused the DB to use more CPU resources as the test progressed.
Database's memory dipped a bit but no symptoms of memory leaks. The memory level bounced back up right after the test finished.
Modules CPU Utilization During Longevity Test
Here is a view of the CPU utilization. A couple of observations:
- mod-authtoken seems to take up CPU resources in a cyclical way from 28% to 32%.
- Okapi uses only about 12%CPU on average compared to Lotus about 450-470% CPU on average.
- mod-users CPU Utilization grows from 20% up to 36%
- Other modules used less than 20% CPU on average.
- mod-circulation-storage spikes periodically due to processing background tasks
Here is the Service CPU Utilization graph with the main involved mods.
There do not appear to be any memory leak issues in Morning Glory. There were no spikes and the processes did not crash.
The timeline highlighted below encompasses all 5 test runs (1, 5, 8, 20, and 50 users).
Modules CPUs and Memory Utilization
The relevant services overall seem to occupy CPU resources nominally. Only mod-authtoken seems to have the spikes but the processes did not crash.
...
but not as aggressive as in Lotus.
| Check In | Check Out | |
|---|---|---|
| 1st Hour | 0.389s | 0.633s |
| 8th Hour | 0.348s | 0.666s |
| 15th Hour | 0.364s | 0.754s |
In the response time graph below the Checkout Controller time, which gathers all check-out API response times), increased over the 16-hours window, from just 0.633s to 0.754s.
The DB CPU utilization percentage did not increase over time and was about10% by the end of the test. There are huge spikes every 30 minutes. These are due to the background tasks that run periodically and as the number of loans grew so did the physical resources needed to process all of them.
The number of connections also rose over time, from 348 to 391 connections. It's unclear what (DB connections) caused the DB to use more CPU resources as the test progressed.
Database's memory dipped a bit but no symptoms of memory leaks. The memory level bounced back up right after the test finished.
Modules CPU Utilization During Longevity Test
Here is a view of the CPU utilization. A couple of observations:
- mod-authtoken seems to take up CPU resources in a cyclical way from 28% to 32%.
- Okapi uses only about 12%CPU on average compared to Lotus about 450-470% CPU on average.
- mod-users CPU Utilization grows from 20% up to 36%
- Other modules used less than 20% CPU on average.
- mod-circulation-storage spikes periodically due to processing background tasks
Here is the Service CPU Utilization graph with the main involved mods.
There do not appear to be any memory leak issues in Morning Glory. There were no spikes and the processes did not crash.
The timeline highlighted below encompasses all 5 test runs (1, 5, 8, 20, and 50 users).
Modules CPUs and Memory Utilization
The relevant services overall seem to occupy CPU resources nominally. Only mod-authtoken seems to have the spikes but the processes did not crash.
| 20-users tests | Avg | Max |
|---|---|---|
| mod-users | 18% | 18% |
| mod-pubsub | 5% | 5% |
| okapi | 11% | 11% |
| mod-circulation | 4% | 4% |
| mod-circulation-storage | 5% | 5% |
| mod-inventory | 6% | 6% |
| mod-inventory-storage | 7% | 7% |
| mod-patron-blocks | 1% | 1% |
| mod-feesfines | 14% | 15% |
| mod-authtoken | 24% | 66% |
| 50-users tests | Avg | Max |
|---|---|---|
| mod-users |
| 38% |
| 39% |
| mod-pubsub |
| 7% |
| 7% |
| okapi |
| 25% |
| 26% |
| mod-circulation |
| 6% |
| 7% |
| mod-circulation-storage |
| 9% |
| 10% | ||
| mod-inventory | 7% | 7% |
| mod-inventory-storage |
| 12% |
| 14% |
| mod-patron-blocks |
| 2% |
| 2% |
| mod-feesfines |
| 31% |
| 33% |
| mod-authtoken |
| 47% |
| 116% |
Services' memory seems to be stable during the test runs.
...
| Users Tested | Morning Glory | Lotus |
|---|---|---|
| 1 | 334 | 350 |
| 5 | 336 | 375 |
| 8 | 346 | 375 |
| 20 | 351 | 400 |
| 50 | 360 | - |
Miscellaneous
- Raw test data: https://docs.google.com/spreadsheets/d/1eVxritKYt5X3g0OO1yaApjLN-2O4iF3CLAnRCAYXdb8/edit#gid=1431274297
...