Master Script stress test - NLA report

Overview

The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors. 

Summary

  • Few test sets was performed:
    • 2,3,6,8 threads tests with xlarge (default) DB;
    • 2,3,6,8 threads tests with 2xlarge DB;
    • test with longer (4 hr) duration for xlarge DB with 8 threads
    • 3 threads test with 4xlarge DB.
  • During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.
  • There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section
  • With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets
  • For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):
    1. ILR_TC: Get ItemId - 709%
    2. VIR_TC: inventory record monitoring - 579%
    3. VTT_TC: Inventory View Instance Source - 531%
    4. VRO_TC: View Vendor - 420%
    5. LO_TC: View Ledgers - 335%
    6. VB_TC: Inventory Choose Instance - 326%
    7. VH_TC: Inventory View Holdings - 322%
    8. VH_TC: Inventory Choose Holdings -308%
    9. ETT_TC: Edit Bib RecordTagTable Group - 304%
    10. VRO_TC: Edit Vendor - 289%


 * - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load

Recommendations & Jiras (Optional)

PERF-525 - Getting issue details... STATUS

  • It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.
  • Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                         "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

    {
                        "name": "JAVA_OPTS",
                        "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
                    },

  • If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

{
    "name": "DB_MAXPOOLSIZE",
    "value": "500"
},
{
    "name": "DB_CONNECTION_TIMEOUT",
    "value": "60"
},

Test Runs 

Test #

Test Conditions

Duration 

Load generator size (recommended)Load generator Memory(GiB) (recommended)

Notes

(Optional)

1.

2 threads Master script60 minst3.xlarge14xlarge DB

2.

3 threads Master script60 minst3.xlarge14xlarge DB
36 threads Master script60 minst3.xlarge14xlarge DB
48 threads Master script60 minst3.xlarge14xlarge DB
52 threads Master script60 minst3.xlarge142xlarge DB
63 threads Master script60 minst3.xlarge142xlarge DB
76 threads Master script60 minst3.xlarge142xlarge DB
88 threads Master script60 minst3.xlarge142xlarge DB
98 threads Master script4 hrt3.xlarge14xlarge DB 
103 threads Master script60 minst3.xlarge144xlarge DB



Results and Comparisons

Performance dependency with increasing load with xlarge DB

Workflow 2 threads response times  3 threads response timesdelta 2 vs 3 threads   6 threads response times delta 3 vs 6 threads 8 threads response times  delta 6 vs 8 threads
AIE_TC: View Invoices Workflow7,1916,991-0,211,6124,62114,2182,606
BLS_TC: Blacklight simulation3,9325,0731,1417,2762,2039,3472,071
ETT_TC: Edit Bib RecordTagTable Group17,05740,93323,87658,7217,78797,4538,73
FYR_TC: Load Finance Ledgers Rollover0,390,5220,1320,5360,0140,7730,237
ILR_TC: Create ILR1,4822,4140,9322,4460,0323,0720,626
ILR_TC: Get ItemId1,731