IN-PROGRESS

...

This is a report for a series of Check-in-check-out test runs against the Juniper Kiwi release. 

BackendBack End:

• mod-circulation-22.1.0
• mod-circulation-storage-13.1.0
• mod-inventory-18.0.0
• mod-inventory-storage-22.0.0
• mod-authtoken-2.9.0
• mod-pubsub-2.4.0
• mod-patron-blocks-1.4.0
• mod-feesfines-17.0.0
• okapi-4.9.0

FrontendFront End:

• folio_circulation-6.0.0
• Item Check-in (folio_checkin-6.0.0)
• Item Check-out (folio_checkout-7.0.0)

EnvironmentInfrastructure:

• 71 back-end modules deployed in 141 ECS tasks
• 3 okapi ECS services
• 6 m5.xlarge  EC2 instances
• 2 db.r6g.xlarge AWS RDS instance
• INFO logging level

...

• Kiwi performs much better than Juniper GA, response times for check-ins are <900ms, checkout <1500ms, with not much variation between 1 and 20 users
• Database performance is better and uses much less CPU compared to Juniper GA
• Worst-performing APIs are still POST /checkin-by-barcode and POST /checkout-by-barcode.  The response time are still about 500ms. GET /circulation/loans also takes more than 200ms. GET /inventory/item (by barcode) takes less than 100ms now.
• Longevity test shows response times worsen over time, probably due to the growing DB CPU utilization. 

  Jira Legacy
  server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key CIRCSTORE-304

  Potentially could address this situation

Test Runs

...

In general there is no regression in performance.  The response times between Kiwi and Juniper are very close to each other for 1-8 users load unless they were in the 95th percentile group or the 20 users load where Kiwi clearly out-perform Juniper.  In the tables below, the Delta columns express the differences between Juniper and Kiwi releases in percentage. Any percentage +/-5% is not statistically is within the margin of error.  It is also noteworthy that Kiwi seems to invoke the GET /automated-patron-blocks 3 times instead of once

Note: JP = Juniper build, KW = Kiwi build

...

Average Response Time in milliseconds. Note: JP = Juniper build, KW = Juniper build 

Longevity

...

Test

Longevity test shows that the response time increased as time went on. 

In the response time graph below the Checkout Controller time, which gathers all chekc-out API response times), increased over a 16-hours window, from <2s to over 5s.

Image Added

The DB CPU utilization percentage also increased over time

...

Gather (cost=1000.00..60093.80 rows=780 width=966) (actual time=192.948..200.814 rows=1 loops=1)
 Workers Planned: 2
 Workers Launched: 2
 -> Parallel Seq Scan on loan (cost=0.00..59015.80 rows=325 width=966) (actual time=145.247..191.854 rows=0 loops=3)
 Filter: ((lower(fs09000000_mod_circulation_storage.f_unaccent((jsonb ->> 'userId'::text))) ~~ '664e4002-9eb2-4c9e-addd-04f308a8062c'::text) AND (lower(fs09000000_mod_circulation_storage.f_unaccent(((jsonb -> 'status'::text) ->> 'name'::text))) !~~ 'closed'::text))
 Rows Removed by Filter: 53759
Planning Time: 0.212 ms
Execution Time: 200.834 ms

Due to the left() function (more details in 

Image Added

Before and after fixing the left() function, there were spikes at half an hour interval, but these were due to background jobs processing the loans (age-to-lost, patron notices, etc..). As more loans got created during the test, the spikes got higher. 

Here is a view of the CPU utilization. A couple of observations:

...

Here is the Service CPU Utilization graph without Okapi for better clarity of other modules' behaviors.

(Note that at around 8:00 (on the graph) there are extended spikes for all the modules, Okapi, and the DB. This is due to tooling. The tool for some reasons added another 20 concurrent users to the test. The results for this period had been discarded.)

There does not appear to be any memory leaks issues in Kiwi

Image Added

Only mod---

Image Removed

Image Removed

Image Removed

A closer look at the CPU graphs without Okapi.

Image Removedinventory-storage seems to have the spikes but but they were not prolonged and the processes did not crash. 

Image Added

Modules CPUs and Memory Utilization

...

In the 8-users tests, mod-inventory spiked to almost 400% while averaged around 175%. This is an anomaly that can be disregarded. We have seen mod-inventory spiked while being idle or during a test run of any workflow before, mostly due to DI events_cache topic processing. In other tests this is not a problem, so it does not seem to be a bad trend here. 

...

Miscellaneous

• Raw test data: 

  View file
  name CICO-kiwi.xlsx height 250