...

This is a report for a series of Check-in-check-out test runs against the Honeysuckle release.

Jira Legacy

server	System JiraJIRA
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-135

...

Check-out: Honeysuckle is slower by 9%-28% than Goldenrod
Check-in: 4%-22% slower than Goldenrod
APIs turned slower in Honeysuckle: GET /automated-patron-blocks/{id} (150% slower) and GET /circulation/loans (60%). These are covered by MODPATBLK-70 and CIRC-1014, respectively
Okapi v4.3.3 seem to be using 2x-3x CPU cycles than in v1.3.2 (Goldenrod). Potential issue found with the logging methods OKAPI-964
mod-pubsub has a memory leak that would drag down performance under high loads (see section on longevity test): MODPUBSUB-136
Caching Okapi tokens in Okapi reduced mod-authtoken's CPU usage by over 90%
Database's memory usage improved dramatically from Goldenrod's - little memory consumptions observed.

...

Subsequent investigations (
Jira Legacy
server System JiraJIRA
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key PERF-140
AND
Jira Legacy
server System JiraJIRA
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key CIRC-1014
) on GET /circulation/loans do not show degradations by the API itself. We hypothesize that other API calls that were executed during the test run may have dragged down the response time, particularly if it was trying to read and write to the same rows in the database at the same time.

...

Honeysuckle tests revealed the following missing indexes:

mod-circulation-storage missing indexes

Code Block

WARNING: Doing LIKE search without index for jsonb->>'requestId', CQL >>> SQL: requestId == 920e1d64-c221-48a0-a44d-ff50f3ad6cd6 >>> lower(f_unaccent(jsonb->>'requestId')) LIKE lower(f_unaccent('920e1d64-c221-48a0-a44d-ff50f3ad6cd6'))
WARNING: Doing FT search without index for request.jsonb->>'requesterId', CQL >>> SQL: requesterId = ae4c1cf3-0738-4465-8112-e75089e5b5c6 >>> get_tsvector(f_unaccent(request.jsonb->>'requesterId')) @@ tsquery_phrase(f_unaccent('ae4c1cf3-0738-4465-8112-e75089e5b5c6'))
WARNING: Doing FT search without index for request.jsonb->>'pickupServicePointId', CQL >>> SQL: pickupServicePointId = 7068e104-aa14-4f30-a8bf-71f71cc15e07 >>> get_tsvector(f_unaccent(request.jsonb->>'pickupServicePointId')) @@ tsquery_phrase(f_unaccent('7068e104-aa14-4f30-a8bf-71f71cc15e07'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'patronId', CQL >>> SQL: patronId == d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6 >>> lower(f_unaccent(patron_action_session.jsonb->>'patronId')) LIKE lower(f_unaccent('d7cabcb2-7431-43ea-a2cc-0dfe5bee17c6'))
WARNING: Doing LIKE search without index for patron_action_session.jsonb->>'actionType', CQL >>> SQL: actionType == Check-out >>> lower(f_unaccent(patron_action_session.jsonb->>'actionType')) LIKE lower(f_unaccent('Check-out'))

...

Results for okapi-4.5.2 for 1,5,8,20 users for 30 minute run. From the response times below, the average Check-out for 20 users is slower. On average 60% slower than okapi-4.3.3.

'+' means performance improvement from okapi-4.3.3

'-' means performance degradation from okapi-4.3.3

For 20 users - 4 requests failed out of 113642

Response Times

	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)
	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out
1 user	0.971	2.072	0.92	1.906	1.013	2.093	1.326	2.905
5 users	1.

003

092 +

2.

114

584 -

0.

925

978 +

1

2.

947

323 +

1.

055

16 +

2.

235

746 +

1.

458

622 +

3

4.

149

021 -
8 users	1.

217

429 -

2

3.

467

057 -

1.

099

285 -

2.

207

747 -

1.

357

62 -

2

3.

648

354 -

1

2.

931

415 -

4

5.

095

079 -

20 users

2

3.

409

073 +

5

7.

213

877 -

2.

141

595 +

4

6.

478

307 +

2

3.

763

411 +

5

8.

682

287 +

4

6.

233

409 +

8

14.

484

703 +

CPUs and Memories

Service CPU Utilization:

...

RDS CPU Utilization is around 50% more compared to okapi-4.3.3

...

Comparison okapi-4.5.2 vs okapi-4.6.1

...

okapi-4.6.1

...

is slower than okapi-4.5.2. Checkin is 3.

...

Response Times

...

66% slower and Checkout is 9.48% slower. See below comparison for 8 Users 30-minute test run.

Image Added

Results for okapi-4.6.1

From the response times below, okapi-4.6.1, checkin-checkout for 1 user is a little slower but for 5, 8, 20 users, checkin-checkout is much faster comparing to okapi-4.3.3.

...

Response Times Okapi-4.3.3

	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)
	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out
1 user	0.94

1

2.

041 -7%

158

0.885

2.

332 -17%

017

0.

957

969

2.

139

177

1.

06

198

2.

369

906
5 users	1.

378

126

3

2.

3945 users1.057 -0.3%2.374-9%0.9782.1761.1332.5321.5243.6248 users1.277 -7%2.814 -25%1.1442.5121.443.0742.1124.71820 users2.374 +0.7%5.927 -51.9%2.1375.2462.7166.5524.1889.426

CPUs and Memories

Service CPU Utilization:

Compared to okapi-4.3.3, CPU Utilization for okapi-4.6.1 has almost doubled! Performance has degraded more than 50%

Image Removed

Service Memory Utilization:

Image Removed

RDS CPU Utilization

Image Removed

Comparison okapi-4.5.2 vs okapi-4.6.1

okapi-4.6.1 is slower than okapi-4.5.2. Checkin is 3.66% slower and Checkout is 9.48% slower. See below comparison for 8 Users 30-minute test run.

Image Removed

8 Hours Longevity test run for 20 Users

Service CPU Utilization:

Image Removed

Service Memory Utilization:

...

574	1.025	2.339	1.211	2.79	1.77	4.007
8 users	1.313	2.948	1.177	2.61	1.487	3.274	2.195	5.045
20 users	3.252	7.492	2.681	6.355	3.605	8.313	7.061	15.747

Response Times Okapi-4.6.1

'+' means a performance improvement from okapi-4.3.3

'-' means a performance degradation from okapi-4.3.3

	Average (seconds)				50th %tile (seconds)		75th %tile (seconds)				95th %tile (seconds)
	Check-in	Check-in performance with okapi-4.3.3	Check-out	Check-out performance with okapi-4.3.3	Check-in	Check-out	Check-in	Check-in performance with okapi-4.3.3	Check-out	Check-out performance with okapi-4.3.3	Check-in	Check-out
1 user	1.041	-9.7%	2.332	-7%	0.957	2.139	1.06	-8.5%	2.369	-8.10%	1.378	3.394
5 users	1.057	+6.5%	2.374	+8.4%	0.978	2.176	1.133	+6.88%	2.532	+10.18%	1.524	3.624
8 users	1.277	+2.8%	2.814	+4.7%	1.144	2.512	1.44	+3.2%	3.074	+6.50%	2.112	4.718
20 users	2.374	+36.9%	5.927	+26.4%	2.137	5.246	2.716	+32.7	6.552	+26.87	4.188	9.426

CPUs and Memories

Service CPU Utilization:

Compared to okapi-4.3.3, CPU Utilization for okapi-4.6.1 around the same.

Image Added

Service Memory Utilization:

Compared to okapi-4.3.3, Service Memory Utilization for okapi-4.6.1 around the same.

Image Added

RDS CPU Utilization

RDS CPU Utilization is normal for 1, 5, and 8 Users. For 20 Users, the CPU is higher almost 95% but considering the large load, it is normal as well.

Image Added

8 Hours Longevity test run for 20 Users

Service CPU Utilization:

Okapi-4.6.1 consumes less CPU gradually from 1st hour into 8th hour. However, at the same time, mod-pubsub consumes more CPU by gradually increasing from 50% to almost 160%

Image Added

Service Memory Utilization:

Memory usage for okapi-4.6.1 and other modules is relatively constant and stable throughout the run. mod-circulation memory consumption increases to 125% and then stabilizes.

Image Added

Comparison okapi-4.3.3 vs okapi-4.6.1 (okapi metrics enabled)

Below results are for 8 Users, 30 minutes against Checkin-Checkout workflow.

Grafana Performance Dashboard

Okapi-4.6.1 is around 71% faster than Okapi-4.3.3. Okapi-4.6.1 can process more requests and still perform better. In 30 minutes test run, okapi-4.6.1 was able to process 25% more requests with an average request per second(RPS) is 40.

Okapi-4.3.3 Grafana performance dashboard:

Image Added

Okapi-4.6.1 Grafana performance dashboard:

Image Added

Checkin-Checkout API level comparison

For Okapi-4.6.1, Check-in is 71% faster and Checkout is around 65% faster.

Image Added

Log request/response comparison

For Okapi-4.6.1, Log request has improved from 3.16 second to 0.243 seconds. Log request is faster by 1200% faster. Log response has improved from 3.25 seconds to 0.266 seconds which is 1100% faster.

Okapi-4.6.1 can process more log request/response and at the same time be fast. Okapi-4.6.1 processed [211k(logRequest), 220k(logResponse)] vs Okapi-4.3.3 which processed [170k(logRequest), 177k(logResponse)].

Okapi-4.3.3 log request/response comparison:

Image Added

Okapi-4.6.1 log request/response comparison:

Image Added

Service CPU Utilization

Okapi-4.6.1 consumes less CPU and hence more efficient. Average CPU Utilization for okapi-4.6.1 is around 380% vs okapi-4.3.3 which is 600%.

Okapi-4.3.3 Service CPU Utilization:

Image Added

Okapi-4.6.1 Service CPU Utilization:

Image Added

Appendix

CIRC-1014

https://issuesfolio-org.folioatlassian.orgnet/browse/MODPATBLK-70

https://issuesfolio-org.folioatlassian.orgnet/browse/OKAPI-964

https://issuesfolio-org.folioatlassian.orgnet/browse/OKAPI-965

checkout-checkin-4.5.2-test-runs

Versions Compared

Old Version 24

New Version Current

Key

CPUs and Memories

Comparison okapi-4.5.2 vs okapi-4.6.1

Image Added

Results for okapi-4.6.1

CPUs and Memories

RDS CPU Utilization

Comparison okapi-4.5.2 vs okapi-4.6.1

Image Removed

8 Hours Longevity test run for 20 Users

CPUs and Memories

RDS CPU Utilization

8 Hours Longevity test run for 20 Users

Comparison okapi-4.3.3 vs okapi-4.6.1 (okapi metrics enabled)

Grafana Performance Dashboard

Checkin-Checkout API level comparison

Log request/response comparison

Service CPU Utilization

Appendix

Page Comparison

Versions Compared

Old Version 24

New Version Current

Key

CPUs and Memories

Comparison okapi-4.5.2 vs okapi-4.6.1

Image Added

Results for okapi-4.6.1

CPUs and Memories

RDS CPU Utilization

Comparison okapi-4.5.2 vs okapi-4.6.1

Image Removed

8 Hours Longevity test run for 20 Users

CPUs and Memories

RDS CPU Utilization

8 Hours Longevity test run for 20 Users

Comparison okapi-4.3.3 vs okapi-4.6.1 (okapi metrics enabled)

Grafana Performance Dashboard

Checkin-Checkout API level comparison

Log request/response comparison

Service CPU Utilization

Appendix