Overview

The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors. 

Summary

• Few test sets was performed:

  • 2,3,6,8 threads tests with xlarge (default) DB;

  • 2,3,6,8 threads tests with 2xlarge DB;

  • test with longer (4 hr) duration for xlarge DB with 8 threads

  • 3 threads test with 4xlarge DB.

• During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.

• There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section

• With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets

• For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):

  1. ILR_TC: Get ItemId - 709%

  2. VIR_TC: inventory record monitoring - 579%

  3. VTT_TC: Inventory View Instance Source - 531%

  4. VRO_TC: View Vendor - 420%

  5. LO_TC: View Ledgers - 335%

  6. VB_TC: Inventory Choose Instance - 326%

  7. VH_TC: Inventory View Holdings - 322%

  8. VH_TC: Inventory Choose Holdings -308%

  9. ETT_TC: Edit Bib RecordTagTable Group - 304%

  10. VRO_TC: Edit Vendor - 289%

 * - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load

Recommendations & Jiras (Optional)

PERF-525: NLA: Improve performance of Fixed load testsClosedPreview

• It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

• Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                       "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

• If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid 

  
  PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

Test Runs 

Results and Comparisons

Performance dependency with increasing load with xlarge DB