Skip to end of banner
Go to start of banner

Export_UUIDS_Test_Results

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Edelweiss (Q4.0 2019) Test Results

Overview

Using the Carrier-io framework for capturing and analyzing performance test results, the following tests for the check-in/check-out workflows were executed.

Test Runs

  • One-hour Runs for checkin/out workflow:

    Test

    Virtual Users

    Duration

    OKAPI log level

    Profiled

    1. Daisy-Baseline

    1

    1 hour

    ERROR

    No

    2. Edelweiss

    1

    1 hour

    ERROR

    No

    3. Daisy-Baseline

    8

    1 hour

    ERROR

    No

    4. Edelweiss

    8

    1 hour

    ERROR

    No

    5. Daisy-Baseline

    20

    1 hour

    ERROR

    No

    6. Edelweiss

    20

    1 hour

    ERROR

    No

    7. Edelweiss

    8

    1 hour

    ERROR

    Yes

    8. Edelweiss Volume Test

    8

    1 hour

    ERROR

    No

    9. Edelweiss Volume Test

    8

    1 hour

    ERROR

    Yes
  • Longevity tests

    Test

    Virtual Users

    Duration

    OKAPI log level

    Profiled

    10. Daisy-Baseline

    8

    8 hours

    ERROR

    No

    11. Edelweiss

    8

    8 hours

    ERROR

    No

    12. Edelweiss - 16 hrs

    8

    16 hours

    ERROR

    No

    Results

    *All numbers are in milliseconds except for those in the Delta % column, which indicates the difference in percentage going from v3.2 to v4.0

    1. High level Edelweiss results data


    The chart shows the overall high-level API stats obtained by JMeter calling various APIs in the checkin/checkout worfklows. It breaks down average response times for 1, 8, and 20 users tests per API call.  A few things to note:
  • the POST methods: circulation/check-out-by-barcode and circulation/check-in-by-barcode response times are above 100 milliseconds, even for 1 user minimum response time, averaging >250ms for 8-users use case. 
  • These checkout and checkin POST methods request rates peak at 2.5 requests/second.
  • Only GET circulation/loans  is above 200ms. Other GET methods are below 100ms. GET inventory/items exceeds 100ms above 75th percentile.

Comparing these results the baseline numbers from Daisy v3.2

The numbers from Daisy v3.2 are a little bit better. The charts below offer a clearer side-by-side comparison for the 1, 8, and 20 users tests runs 

  1. Comparing Tests # 1 and #2: 1 user for 1 hour tests '
  2. Comparing Tests #3 and #4: 8 users for 1 hour tests
  3. Comparing Tests #5 and #6: 20 users for 1 hour tests
  4. Comparing Tests  #10 and #11: 8 users for 8 hours longevity tests

2.  CPU Utilization comparisons

D = Daisy, E = Edelweiss These services for the selected modules were chosen for their activity in the workflow and prominent values compared to other modules.
Data were obtained from the 1-hour test runs.
Nothing conclusive is seen from CPU utilization perspective. No increasing or decreasing CPU utilization

3.  Longevity tests: Memory trends

Memory trends were obtained from longevity tests

  1. Okapi exhibits a memory leak over 8 and 16 hours test runs
  2. mod-circulation-storage exhibits a memory leak over 8 and 16 hours test runs
  3. mod-inventory-storage exhibits a memory leak over 8 and 16 hours test runs
  4. mod-circulation shows a small leak
  5. Other modules showed very small increases over the longevity test runs duration (1%-2%)

4.  Longevity tests: Database CPU trends  

Improvement in CPU trends over 8 hours of testing

Compared to q3.2 Daisy 8 hours run:

5.  Volume Test comparisons


 DefDS = Default dataset (227K inventory records)
 vol DS = Dataset created for volume test. 227K inventory records with 113.5K loan data
Results show a huge response time in the presence of lots of loan data. This is due to a slow query identified in previous tests: 

Summary

        See Attached Edelweiss Performance Test Runs.xlsx for details

    • FOLIO performs 2x better with ERROR log level than with INFO log level.
    • FOLIO performs better without being profiled when the tests are running 

Issues


    • POST_circulation/check-out-by-barcode and POST_circulation/check-in-by-barcode perform about 30%-35% slower. Generally all methods performed slower, but not sure if it's because of these two POST requests.
    • POST_circulation/check-out-by-barcode and POST_circulation/check-in-by-barcode average response times are still over 300ms in the 8-users tests
    • Memory Issues: 
      • Okapi, mod-circulation-storage, mod-users have noticeable to significant gains in memory used over time.  (Not new)
      • Memory Issues: most modules, if not all, trend up over time by about 1%-2%.
    • org.slf4j.impl.Log4jLoggerAdapter.isDebugEnabled is still among the slowest methods, after jackson deserialization

Improvements


    • There's an improvement in GET_circulation/loans response time
    • Memory Issues: mod-inventory-storage OOM seemed to have been fixed in Edelweiss. 3% rise over 8 hours instead of OOM previously
    • Q3.2 mod-inventory-storage OOM

OKAPI-2.37.0-SNAPSHOT

OKAPI-795|796 Improvements





  • No labels