IN-PROGRESS
Table of Contents | ||
---|---|---|
|
...
This is a report for a series of Check-in-check-out test runs against the Juniper Kiwi release.
BackendBack End:
- mod-circulation-22.1.0
- mod-circulation-storage-13.1.0
- mod-inventory-18.0.0
- mod-inventory-storage-22.0.0
- mod-authtoken-2.9.0
- mod-pubsub-2.4.0
- mod-patron-blocks-1.4.0
- mod-feesfines-17.0.0
- okapi-4.9.0
FrontendFront End:
- folio_circulation-6.0.0
- Item Check-in (folio_checkin-6.0.0)
- Item Check-out (folio_checkout-7.0.0)
EnvironmentInfrastructure:
- 71 back-end modules deployed in 141 ECS tasks
- 3 okapi ECS services
- 6 m5.xlarge EC2 instances
- 2 db.r6g.xlarge AWS RDS instance
- INFO logging level
...
- Kiwi performs much better than Juniper GA, response times for check-ins are <900ms, checkout <1500ms, with not much variation between 1 and 20 users
- Database performance is better and uses much less CPU compared to Juniper GA
- Worst-performing APIs are still POST /checkin-by-barcode and POST /checkout-by-barcode. The response time are still about 500ms. GET /circulation/loans also takes more than 200ms. GET /inventory/item (by barcode) takes less than 100ms now.
- Longevity test shows response times worsen over time, probably due to the growing DB CPU utilization.
Potentially could address this situationJira Legacy server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key CIRCSTORE-304
Test Runs
Test | Virtual Users | Duration |
1. | 1 | 30 mins |
2. | 5 | 30 mins |
3. | 8 | 30 mins |
4. | 20 | 30 mins |
5. | 1 (repeat) | 30 mins |
...
In general there is no regression in performance. The response times between Kiwi and Juniper are very close to each other for 1-8 users load unless they were in the 95th percentile group or the 20 users load where Kiwi clearly out-perform Juniper. In the tables below, the Delta columns express the differences between Juniper and Kiwi releases in percentage. Any percentage +/-5% is not statistically is within the margin of error. It is also noteworthy that Kiwi seems to invoke the GET /automated-patron-blocks 3 times instead of once
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Note: JP = Juniper build, KW = Kiwi build
Average | 50th percentile | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Check-in JP | Check-in KW | Delta | Check-out JP | Check-out KW | Delta | Check-in JP | Check-in KW | Delta | Check-out JP | Check-out KW | Delta | |
1 user | 0.944 | 0.838 | 11.23% | 1.579 | 1.582 | -0.19% | 0.835 | 0.767 | 8.14% | 1.411 | 1.464 | -3.76% |
5 users | 0.811 | 0.73 | 9.99% | 1.359 | 1.376 | -1.25% | 0.750 | 0.676 | 9.87% | 1.23 | 1.272 | -3.41% |
8 users | 0.889 | 0.758 | 14.74% | 1.425 | 1.392 | 2.32% | 0.785 | 0.674 | 14.14% | 1.262 | 1.228 | 2.69% |
20 users | 1.386 | 0.899 | 35.14% | 2.21 | 1.506 | 31.86% | 1.172 | 0.731 | 37.63% | 1.887 | 1.303 | 30.95% |
...
Average Response Time in milliseconds. Note: JP = Juniper build, KW = Juniper build
API | 1 user JP (75th %tile) | 1 user KW (75th %tile) | 5 users JP (75th %tile) | 5 users KW (75th %tile) | 8 users JP (75th %tile) | 8 users KW (75th %tile) | 20 users JP (75th %tile) | 20 users KW (75th %tile) |
---|---|---|---|---|---|---|---|---|
POST checkout-by-barcode | 526 | 550 | 476 | 436 | 479 | 428 | 861 | 461 |
POST checkin-by-barcode | 457 | 444 | 402 | 368 | 489 | 438 | 852 | 504 |
GET circulation/loans | 246 | 305 | 245 | 262 | 272 | 267 | 516 | 303 |
GET inventory/items | 171 | 124 | 166 | 96 | 170 | 92 | 250 | 99 |
Longevity
...
Test
Longevity test shows that the response time increased as time went on.
Check In | Check Out | |
---|---|---|
1st Hour | 0.821s | 1.476s |
16th Hour | 1.129s | 3.343s |
25th Hour | 0.999s | 3.758s |
In the response time graph below the Checkout Controller time, which gathers all chekc-out API response times), increased over a 16-hours window, from <2s to over 5s.
The DB CPU utilization percentage also increased over time
...
Doing an EXPLAIN ANALYZE on the query shows that it was not using indexes in the query plan despite having the indexes to look up loans by userId and loan status name.
Code Block |
---|
Gather (cost=1000.00..60093.80 rows=780 width=966) (actual time=192.948..200.814 rows=1 loops=1) Workers Planned: 2 Workers Launched: 2 -> Parallel Seq Scan on loan (cost=0.00..59015.80 rows=325 width=966) (actual time=145.247..191.854 rows=0 loops=3) Filter: ((lower(fs09000000_mod_circulation_storage.f_unaccent((jsonb ->> 'userId'::text))) ~~ '664e4002-9eb2-4c9e-addd-04f308a8062c'::text) AND (lower(fs09000000_mod_circulation_storage.f_unaccent(((jsonb -> 'status'::text) ->> 'name'::text))) !~~ 'closed'::text)) Rows Removed by Filter: 53759 Planning Time: 0.212 ms Execution Time: 200.834 ms |
Due to the left() function (more details in
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Before and after fixing the left() function, there were spikes at half an hour interval, but these were due to background jobs processing the loans (age-to-lost, patron notices, etc..). As more loans got created during the test, the spikes got higher.
Here is a view of the CPU utilization. A couple of observations:
...
Here is the Service CPU Utilization graph without Okapi for better clarity of other modules' behaviors.
(Note that at around 8:00 (on the graph) there are extended spikes for all the modules, Okapi, and the DB. This is due to tooling. The tool for some reasons added another 20 concurrent users to the test. The results for this period had been discarded.)
There does not appear to be any memory leaks issues in Kiwi
Only mod---
A closer look at the CPU graphs without Okapi.
inventory-storage seems to have the spikes but but they were not prolonged and the processes did not crash.
Modules CPUs and Memory Utilization
...
In the 8-users tests, mod-inventory spiked to almost 400% while averaged around 175%. This is an anomaly that can be disregarded. We have seen mod-inventory spiked while being idle or during a test run of any workflow before, mostly due to DI events_cache topic processing. In other tests this is not a problem, so it does not seem to be a bad trend here.
...
Miscellaneous
- Raw test data:
View file name CICO-kiwi.xlsx height 250