Overview

The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors. 

Summary

• Few test sets was performed:
  • 2,3,6,8 threads tests with xlarge (default) DB;
  • 2,3,6,8 threads tests with 2xlarge DB;
  • test with longer (4 hr) duration for xlarge DB with 8 threads
  • 3 threads test with 4xlarge DB.
• During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.
• There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section
• With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets
• For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):
  1. ILR_TC: Get ItemId - 709%
  2. VIR_TC: inventory record monitoring - 579%
  3. VTT_TC: Inventory View Instance Source - 531%
  4. VRO_TC: View Vendor - 420%
  5. LO_TC: View Ledgers - 335%
  6. VB_TC: Inventory Choose Instance - 326%
  7. VH_TC: Inventory View Holdings - 322%
  8. VH_TC: Inventory Choose Holdings -308%
  9. ETT_TC: Edit Bib RecordTagTable Group - 304%
  10. VRO_TC: Edit Vendor - 289%

 * - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load

Recommendations & Jiras (Optional)

PERF-525 - Getting issue details... STATUS

• It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

• Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                       "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

  {                     "name": "JAVA_OPTS",                     "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"                 },

• If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

Test Runs 

Results and Comparisons

Performance dependency with increasing load with xlarge DB