Edelweiss (Q4.0 2019) Test Results

• Overview
• Test Runs
• Results
  • 1. High level Edelweiss results data
  • 2.  CPU Utilization comparisons
  • 3.  Longevity tests: Memory trends
  • 4.  Longevity tests: Database CPU trends  
  • 5.  Volume Test comparisons
• Summary
  • Issues
  • Improvements
• OKAPI-2.37.0-SNAPSHOT

Overview

Using the Carrier-io framework for capturing and analyzing performance test results, the following tests for the check-in/check-out workflows were executed.

Test Runs

• One-hour Runs for checkin/out workflow:

  Test	Virtual Users	Duration	OKAPI log level	Profiled
  1. Daisy-Baseline	1	1 hour	ERROR	No
  2. Edelweiss	1	1 hour	ERROR	No
  3. Daisy-Baseline	8	1 hour	ERROR	No
  4. Edelweiss	8	1 hour	ERROR	No
  5. Daisy-Baseline	20	1 hour	ERROR	No
  6. Edelweiss	20	1 hour	ERROR	No
  7. Edelweiss	8	1 hour	ERROR	Yes
  8. Edelweiss Volume Test	8	1 hour	ERROR	No
  9. Edelweiss Volume Test	8	1 hour	ERROR	Yes

• Longevity tests

  Test	Virtual Users	Duration	OKAPI log level	Profiled
  10. Daisy-Baseline	8	8 hours	ERROR	No
  11. Edelweiss	8	8 hours	ERROR	No
  12. Edelweiss - 16 hrs	8	16 hours	ERROR	No

  Results

  *All numbers are in milliseconds except for those in the Delta % column, which indicates the difference in percentage going from v3.2 to v4.0

  1. High level Edelweiss results data

  Image Added
  The chart shows the overall high-level API stats obtained by JMeter calling various APIs in the checkin/checkout worfklows. It breaks down average response times for 1, 8, and 20 users tests per API call.  A few things to note:
• the POST methods: circulation/check-out-by-barcode and circulation/check-in-by-barcode response times are above 100 milliseconds, even for 1 user minimum response time, averaging >250ms for 8-users use case. 
• These checkout and checkin POST methods request rates peak at 2.5 requests/second.
• Only GET circulation/loans  is above 200ms. Other GET methods are below 100ms. GET inventory/items exceeds 100ms above 75th percentile.

Comparing these results the baseline numbers from Daisy v3.2
Image Added
The numbers from Daisy v3.2 are a little bit better. The charts below offer a clearer side-by-side comparison for the 1, 8, and 20 users tests runs 

1. Comparing Tests # 1 and #2: 1 user for 1 hour tests Image Added '
2. Comparing Tests #3 and #4: 8 users for 1 hour tests Image Added
3. Comparing Tests #5 and #6: 20 users for 1 hour tests Image Added
4. Comparing Tests  #10 and #11: 8 users for 8 hours longevity tests Image Added

2.  CPU Utilization comparisons

Image Added D = Daisy, E = Edelweiss These services for the selected modules were chosen for their activity in the workflow and prominent values compared to other modules.
Data were obtained from the 1-hour test runs.
Nothing conclusive is seen from CPU utilization perspective. No increasing or decreasing CPU utilization

3.  Longevity tests: Memory trends

Memory trends were obtained from longevity tests

1. Okapi exhibits a memory leak over 8 and 16 hours test runs Image Added
2. mod-circulation-storage exhibits a memory leak over 8 and 16 hours test runs Image Added
3. mod-inventory-storage exhibits a memory leak over 8 and 16 hours test runs Image Added
4. mod-circulation shows a small leak Image Added
5. Other modules showed very small increases over the longevity test runs duration (1%-2%)

4.  Longevity tests: Database CPU trends  

Improvement in CPU trends over 8 hours of testing
Image Added
Compared to q3.2 Daisy 8 hours run:
Image Added

5.  Volume Test comparisons

Image Added
 DefDS = Default dataset (227K inventory records)
 vol DS = Dataset created for volume test. 227K inventory records with 113.5K loan data
Results show a huge response time in the presence of lots of loan data. This is due to a slow query identified in previous tests: 
Image Added

Summary

        See Attached Edelweiss Performance Test Runs.xlsx for details

• • FOLIO performs 2x better with ERROR log level than with INFO log level.
  • FOLIO performs better without being profiled when the tests are running 

Issues

• • POST_circulation/check-out-by-barcode and POST_circulation/check-in-by-barcode perform about 30%-35% slower. Generally all methods performed slower, but not sure if it's because of these two POST requests.
  • POST_circulation/check-out-by-barcode and POST_circulation/check-in-by-barcode average response times are still over 300ms in the 8-users tests
  • Memory Issues: 
    • Okapi, mod-circulation-storage, mod-users have noticeable to significant gains in memory used over time.  (Not new)
    • Memory Issues: most modules, if not all, trend up over time by about 1%-2%.
  • org.slf4j.impl.Log4jLoggerAdapter.isDebugEnabled is still among the slowest methods, after jackson deserialization

Improvements

• • There's an improvement in GET_circulation/loans response time
  • Memory Issues: mod-inventory-storage OOM seemed to have been fixed in Edelweiss. 3% rise over 8 hours instead of OOM previously Image Added
  • Q3.2 mod-inventory-storage OOM Image Added Image Added

OKAPI-2.37.0-SNAPSHOT

OKAPI-795|796 Improvements
Image Added
Image Added
Image Added