...

https://issues.folio.org/browse/OKAPI-965

IN-REVIEW

Overview

Backend:

• mod-circulation-19.2.7
• mod-circulation-storage-12.1.4
• mod-inventory-16.1.3
• mod-inventory-storage-19.4.4
• mod-authtoken-2.6.0
• mod-pubsub-1.3.3
• okapi-4.3.3 (also with 4.2.2)

Frontend:

• folio_circulation-4.0.1
• Item Check-in (folio_checkin-4.0.1)
• Item Check-out (folio_checkout-5.0.1)

Environment:

• 61 back-end modules deployed in 110 ECS services
• 3 okapi ECS services
• 8 m5.large  EC2 instances
• 2 db.r5.xlarge AWS RDS instance
• INFO logging level

High Level Summary

• Check-out: Honeysuckle is slower by 9%-28% than Goldenrod
• Check-in: 4%-22% slower than Goldenrod
• APIs turned slower in Honeysuckle: GET /automated-patron-blocks/{id} (150% slower) and  GET /circulation/loans (60%). These are covered by MODPATBLK-70 and CIRC-1014, respectively
• Okapi v4.3.3 seem to be using 2x-3x CPU cycles than in v1.3.2 (Goldenrod).  Potential issue found with the logging methods OKAPI-964 
• mod-pubsub has a memory leak that would drag down performance under high loads (see section on longevity test): MODPUBSUB-136
• Caching Okapi tokens in Okapi reduced mod-authtoken's CPU usage by over 90%
• Database's memory usage improved dramatically from Goldenrod's - little memory consumptions observed.

Test Runs

...

Test

...

Virtual Users

...

Duration

...

OKAPI log level

...

1.

...

2.

...

3.

...

4.

...

5.

...

Results

Response Times

...

The following table shows the slow 75th percentile APIs taking more than 100 ms to return, and also comparing them against Goldenrod's.  Other than the 1-user test, starting with the 5-users test, all listed APIs are slower with GET automated-patron-blocks leading the way at 150% slower, while GET circulation/loans regressed up to 60%

...

API

...

1 user  HS (75th %tile)

...

5 users GR (75th %tile)

...

5 users HS (75th %tile)

...

20 users GR (75th %tile)

...

POST checkin-by-barcode 

...

This table shows that the average response time of the checkout transaction is 9% to 28% slower than in Goldenrod, and check-in in Honeysuckle likewise is about 4-22% slower (data aggregated from 5 to 20 users test runs).

Longevity test

A 24-hour longevity test was performed with 20 concurrent users and running with Okapi 4.2.2. Key observations:

Image Removed

• Performance degraded after 2 hours.
  • Average response time in the first two hours: 
    • Check-in:  2.28 sec
    • Check-out: 4.76 sec
  • Average response time in the first 12 hours:
    • Check-in: 2.709 sec
    • Check-out: 7.898 sec
  • Average response time in the last 12 hours:
    • Check-in:  4.605 sec
    • Check-out: 17.297 sec
  • Together the average response time for the entire test run:
    • Check-in: 3.414 sec
    • Check-out: 11.419 sec
• Throughput degrades over time, unsurprisingly
  • Image Removed

• Modules Services CPU utilizations (The black lines indicate the start and finish time of the test run)
• Image Removed
  • With the load of 20 users, Okapi started out working hardest, but its CPU utilization drops down over time. This is because mod-pubsub's is leaking HTTP Client and taking up more resources therefore slowing down, causing Okapi to not need to route messages as much. Clearly mod-pubsub's CPU utilization went up. The other modules follow Okapi's pattern.

• Services Modules Memory utilizations 
• Image Removed
  • No modules exhibited memory leaks except for mod-pubsub
• Although there were two instances of mod-pubsub running on two different ec2 instances, mod-pubsub's traffic seemed to have been stickied to one instance. Here are graphs showing mod-pubsub's on one instance using up memory and CPU resources, and on another instance not showing much activities:
  • Image Removed
  • mod-pubsub and Okapi on another node - Okapi's CPU utilization dwindles while mod-pubsub does not seem to be busy at all
  • Image Removed

CPUs and Memories

Okapi was profiled because of the apparent 3x CPU utilization compared to the Goldenrod runs.

Goldenrod test runs:

Image Removed

Honeysuckle test runs:

Image Removed

Clearly Okapi is using more CPU cycles in Honeysuckle than in Goldenrod, 1 user 

Other relevant modules' CPU utilizations in Goldenrod

Image Removed

Same modules in Honeysuckle:

Image Removed

A few things to note:

1. mod-authtoken uses much less CPU in Honeysuckle, over 90% reduction across all tests! This is because of the token caching functionality that was added to Okapi 4.x
2. mod-circulation's CPU utilization in Honeysuckle averages over 20% lower than in Goldenrod.
3. mod-circulation's CPU utilization in Honeysuckle is about 10-30% higher than in Goldenrod
4. mod-inventory's CPU utilization in Honeysuckle averages 30% more than in Goldenrod
5. mod-inventory-storage's CPU utilization in Honeysuckle averages 20% more than in Goldenrod 
6. mod-pubsub's CPU utilization in Honeysuckle is about 15% less than in Goldenrod
7. mod-patron-blocks CPU utilization in Honeysuckle is at least 30% less than in Goldenrod

JVM Profiling

Because Okapi's CPU utilization in Honeysuckle seemed to have averaged 2x to 3x higher than in Goldenrod, it was profiled to get more insights of what happened inside it.

The slowest methods in Honeysuckle are once again the Logger and Jackson serialization methods

Image Removed

Compared to Goldenrod:

Image Removed

Note that the AbstractLogger.Info method in Okapi 4.3.3 total CPU time is about 3x higher than in Goldenrod.  This is confirmed by Okapi 4.3.3's metrics showing ProxyContext.logRequest and ProxyContext.logResponse methods' response times degrade over time. These two methods need to be investigated.

Image Removed

Image Removed

Database

The database CPU utilizations are about the same between the Honeysuckle and Goldenrod

Honeysuckle's 

Image Removed

Goldenrod's

Image Removed

Honeysuckle's database memory utilization is much much better than Goldenrod's.  For the most part Honeysuckle's consecutive test runs did not produce signs of aggressive memory leaks that were seen in Goldenrod. 

Image Removed

Goldenrod's memory profile shows quick claims of memory over 30 minutes tests runs. 

Image Removed

Missing Indexes

Honeysuckle tests revealed the following missing indexes:

mod-circulation missing indexes

WARNING: Doing LIKE search without index for jsonb->>'requestId', CQL >>> SQL: requestId == 920e1d64-c221-48a0-a44d-ff50f3ad6cd6 >>> lower(f_unaccent(jsonb->>'requestId')) LIKE lower(f_unaccent('920e1d64-c221-48a0-a44d-ff50f3ad6cd6'))
WARNING: Doing FT search without index for request.jsonb->>'requesterId', CQL >>> SQL: requesterId = ae4c1cf3-0738-4465-8112-e75089e5b5c6 >>> get_tsvector(f_unaccent(request.jsonb->>'requesterId')) @@ tsquery_phrase(f_unaccent('ae4c1cf3-0738-4465-8112-e75089e5b5c6'))
WARNING: Doing FT search without index for request.jsonb->>'pickupServicePointId', CQL >>> SQL: pickupServicePointId = 7068e104-aa14-4f30-a8bf-71f71cc15e07 >>> get_tsvector(f_unaccent(request.jsonb->>'pickupServicePointId')) @@ tsquery_phrase(f_unaccent('7068e104-aa14-4f30-a8bf-71f71cc15e07'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'patronId', CQL >>> SQL: patronId == d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6 >>> lower(f_unaccent(patron_action_session.jsonb->>'patronId')) LIKE lower(f_unaccent('d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'actionType', CQL >>> SQL: actionType == Check-out >>> lower(f_unaccent(patron_action_session.jsonb->>'actionType')) LIKE lower(f_unaccent('Check-out'))

The following warnings are captured when the background tasks are running

WARNING: Doing LIKE search without index for jsonb->'noticeConfig'>>'timing', CQL >>> SQL: noticeConfig.timing == After >>> lower(f_unaccent(jsonb>'noticeConfig'->>'timing')) LIKE lower(f_unaccent('After'))
WARNING: Doing LIKE search without index for jsonb->>'loanId', CQL >>> SQL: loanId == 671233fd-5c15-4f9e-8fab-f86330c389bd >>> lower(f_unaccent(jsonb->>'loanId')) LIKE lower(f_unaccent('671233fd-5c15-4f9e-8fab-f86330c389bd'))
WARNING: Doing LIKE search without index for jsonb->>'triggeringEvent', CQL >>> SQL: triggeringEvent == "Due date" >>> lower(f_unaccent(jsonb->>'triggeringEvent')) LIKE lower(f_unaccent('Due date'))

mod-feesfines missing indexes:

WARNING: Doing LIKE search without index for accounts.jsonb->>'userId', CQL >>> SQL: userId == e96618a9-04ee-4fea-aa60-306a8f4dd89b >>> lower(f_unaccent(accounts.jsonb->>'userId')) LIKE lower(f_unaccent('e96618a9-04ee-4fea-aa60-306a8f4dd89b'))
WARNING: Doing LIKE search without index for accounts.jsonb->'status'>>'name', CQL >>> SQL: status.name <> Closed >>> lower(f_unaccent(accounts.jsonb>'status'->>'name')) NOT LIKE lower(f_unaccent('Closed'))
WARNING: Doing LIKE search without index for manualblocks.jsonb->>'userId', CQL >>> SQL: userId == a79b533d-8f29-4be1-9415-5f5cd936623b >>> lower(f_unaccent(manualblocks.jsonb->>'userId')) LIKE lower(f_unaccent('a79b533d-8f29-4be1-9415-5f5cd936623b'))

Appendix

CIRC-1014

https://issues.folio.org/browse/MODPATBLK-70

https://issues.folio.org/browse/OKAPI-964

https://issues.folio.org/browse/OKAPI-965