Versions Compared

Old Version 27

changes.mady.by.user Varun Javalkar

Saved on

compared with

New Version Current

changes.mady.by.user Varun Javalkar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This is a report for a series of Check-in-check-out test runs against the Honeysuckle release.

Jira Legacy
serverSystem JiraJIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-135

...

  • Check-out: Honeysuckle is slower by 9%-28% than Goldenrod
  • Check-in: 4%-22% slower than Goldenrod
  • APIs turned slower in Honeysuckle: GET /automated-patron-blocks/{id} (150% slower) and  GET /circulation/loans (60%). These are covered by MODPATBLK-70 and CIRC-1014, respectively
  • Okapi v4.3.3 seem to be using 2x-3x CPU cycles than in v1.3.2 (Goldenrod).  Potential issue found with the logging methods OKAPI-964 
  • mod-pubsub has a memory leak that would drag down performance under high loads (see section on longevity test): MODPUBSUB-136
  • Caching Okapi tokens in Okapi reduced mod-authtoken's CPU usage by over 90%
  • Database's memory usage improved dramatically from Goldenrod's - little memory consumptions observed.

...

  • Subsequent investigations (
    Jira Legacy
    serverSystem JiraJIRA
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-140
    AND
    Jira Legacy
    serverSystem JiraJIRA
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyCIRC-1014
    ) on GET /circulation/loans do not show degradations by the API itself. We hypothesize that other API calls that were executed during the test run may have dragged down the response time, particularly if it was trying to read and write to the same rows in the database at the same time. 

...

Honeysuckle tests revealed the following missing indexes:

mod-circulation-storage missing indexes

Code Block
WARNING: Doing LIKE search without index for jsonb->>'requestId', CQL >>> SQL: requestId == 920e1d64-c221-48a0-a44d-ff50f3ad6cd6 >>> lower(f_unaccent(jsonb->>'requestId')) LIKE lower(f_unaccent('920e1d64-c221-48a0-a44d-ff50f3ad6cd6'))
WARNING: Doing FT search without index for request.jsonb->>'requesterId', CQL >>> SQL: requesterId = ae4c1cf3-0738-4465-8112-e75089e5b5c6 >>> get_tsvector(f_unaccent(request.jsonb->>'requesterId')) @@ tsquery_phrase(f_unaccent('ae4c1cf3-0738-4465-8112-e75089e5b5c6'))
WARNING: Doing FT search without index for request.jsonb->>'pickupServicePointId', CQL >>> SQL: pickupServicePointId = 7068e104-aa14-4f30-a8bf-71f71cc15e07 >>> get_tsvector(f_unaccent(request.jsonb->>'pickupServicePointId')) @@ tsquery_phrase(f_unaccent('7068e104-aa14-4f30-a8bf-71f71cc15e07'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'patronId', CQL >>> SQL: patronId == d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6 >>> lower(f_unaccent(patron_action_session.jsonb->>'patronId')) LIKE lower(f_unaccent('d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'actionType', CQL >>> SQL: actionType == Check-out >>> lower(f_unaccent(patron_action_session.jsonb->>'actionType')) LIKE lower(f_unaccent('Check-out'))

...

Results for okapi-4.5.2 for 1,5,8,20 users for 30 minute run. From the response times below, the average Check-out for 20 users is slower. On average 60% slower than okapi-4.3.3.

'+' means performance improvement from okapi-4.3.3

'-' means performance degradation from okapi-4.3.3

For 20 users - 4 requests failed out of 113642

Response Times


Average (seconds)50th %tile (seconds)75th %tile (seconds)95th %tile  (seconds)

Check-inCheck-outCheck-inCheck-outCheck-inCheck-outCheck-inCheck-out
1 user0.9712.0720.921.9061.0132.0931.3262.905
5 users1.
003
092 +2.
114
584 -0.
925
978 +
1
2.
947
323 +1.
055
16 +2.
235
746 +1.
458
622 +
3
4.
149
021 -
8 users1.
217
429 -
2
3.
467
057 -1.
099
285 -2.
207
747 -1.
357
62 -
2
3.
648
354 -
1
2.
931
415 -
4
5.
095
079 -
20 users
2
3.
409
073 +
5
7.
213
877 -2.
141
595 +
4
6.
478
307 +
2
3.
763
411 +
5
8.
682
287 +
4
6.
233
409 +
8
14.
484
703 +


CPUs and Memories

Service CPU Utilization:

...

RDS CPU Utilization is around 50% more compared to okapi-4.3.3

...

Comparison okapi-4.

...

5.2 vs okapi-4.6.1

...

okapi-4.

...

6.1 is slower than okapi-4.

...

5.2.

...

Checkin is 3.66% slower and Checkout is 9.48% slower. See below comparison for 8 Users 30-minute test run.

Image Added


Results for okapi-4.6.1

From the response times below, okapi-4.6.1, checkin-checkout for 1 user is a little slower but for 5, 8, 20 users, checkin-checkout is much faster comparing to okapi-4.3.3.

Response Times Okapi-4.3.3


Average (seconds)50th %tile (seconds)75th %tile (seconds)95th %tile  (seconds)

Check-inCheck-outCheck-inCheck-outCheck-inCheck-outCheck-inCheck-out
1 user0.942.1580.8852.0170.9692.1771.1982.906
5 users1.1262.5741.0252.3391.2112.791.774.007
8 users1.313 2.9481.1772.611.4873.2742.1955.045
20 users3.2527.4922.6816.3553.6058.3137.06115.747

Response Times Okapi-4.6.1

'+' means a performance improvement from okapi-4.3.3

'-' means a performance degradation from okapi-4.3.3


Average (seconds)50th %tile (seconds)75th %tile (seconds)95th %tile  (seconds)

Check-inCheck-in performance with okapi-4.3.3Check-outCheck-out performance with okapi-4.3.3Check-inCheck-outCheck-inCheck-in performance with okapi-4.3.3Check-outCheck-out performance with okapi-4.3.3
Check-inCheck-outCheck-in
Check-
outCheck-
inCheck-out
1 user1.041 -9.7%2.332 -
17%
7%0.9572.1391.06-8.5%2.369-8.10%1.3783.394
5 users1.057 
-0
+6.
3%
5%2.374
-9%
+8.4%0.9782.1761.133+6.88%2.532+10.18%1.5243.624
8 users1.277 
-7%
+2.8%2.814 
-25%
+4.7%1.1442.5121.44+3.2%3.074+6.50%2.1124.718
20 users2.374 +
0
36.
7%
9%5.927 
-51
+26.
9%
4%2.1375.2462.716+32.76.552+26.874.1889.426


CPUs and Memories

Service CPU Utilization:

Compared to okapi-4.3.3, CPU Utilization for okapi-4.6.1 has almost doubled! Performance has degraded more than 50%

Image Removed

Service Memory Utilization:

Image Removed

RDS CPU Utilization

Image Removed

...

okapi-4.

...

3.3, CPU Utilization for okapi-4.6.1 around the same.

Image Added


Service Memory Utilization:

Compared to okapi-4.6.1 is slower than 3.3, Service Memory Utilization for okapi-4.5.2. Checkin is 3.66% slower and Checkout is 9.48% slower. See below comparison for 8 Users 30-minute test run.

...

6.1 around the same.

Image Added


RDS CPU Utilization

RDS CPU Utilization is normal for 1, 5, and 8 Users. For 20 Users, the CPU is higher almost 95% but considering the large load, it is normal as well. 

Image Added

8 Hours Longevity test run for 20 Users

Service CPU Utilization:CPU Utilization:

Okapi-4.6.1 consumes less CPU gradually from 1st hour into 8th hour. However, at the same time, mod-pubsub consumes more CPU by gradually increasing from 50% to almost 160% 


Service Memory Utilization:

Memory usage for okapi-4.6.1  and other modules is relatively constant and stable throughout the run. mod-circulation memory consumption increases to 125% and then stabilizes.


Comparison okapi-4.3.3 vs okapi-4.6.1 (okapi metrics enabled)

...

Okapi-4.6.1 Service CPU Utilization:

Appendix

CIRC-1014

https://issuesfolio-org.folioatlassian.orgnet/browse/MODPATBLK-70

https://issuesfolio-org.folioatlassian.orgnet/browse/OKAPI-964

https://issuesfolio-org.folioatlassian.orgnet/browse/OKAPI-965

checkout-checkin-4.5.2-test-runs