Master Script stress test - NLA report
Overview
The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors.
Summary
- Few test sets was performed:
- 2,3,6,8 threads tests with xlarge (default) DB;
- 2,3,6,8 threads tests with 2xlarge DB;
- test with longer (4 hr) duration for xlarge DB with 8 threads
- 3 threads test with 4xlarge DB.
- During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.
- There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section
- With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets
- For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):
- ILR_TC: Get ItemId - 709%
- VIR_TC: inventory record monitoring - 579%
- VTT_TC: Inventory View Instance Source - 531%
- VRO_TC: View Vendor - 420%
- LO_TC: View Ledgers - 335%
- VB_TC: Inventory Choose Instance - 326%
- VH_TC: Inventory View Holdings - 322%
- VH_TC: Inventory Choose Holdings -308%
- ETT_TC: Edit Bib RecordTagTable Group - 304%
- VRO_TC: Edit Vendor - 289%
* - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load
Recommendations & Jiras (Optional)
- PERF-525Getting issue details... STATUS
- It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.
Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
"-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"{
"name"
:
"JAVA_OPTS"
,
"value"
:
"-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
},
- If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :
|
Test Runs
Test # | Test Conditions | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) | Notes (Optional) |
1. | 2 threads Master script | 60 mins | t3.xlarge | 14 | xlarge DB |
2. | 3 threads Master script | 60 mins | t3.xlarge | 14 | xlarge DB |
3 | 6 threads Master script | 60 mins | t3.xlarge | 14 | xlarge DB |
4 | 8 threads Master script | 60 mins | t3.xlarge | 14 | xlarge DB |
5 | 2 threads Master script | 60 mins | t3.xlarge | 14 | 2xlarge DB |
6 | 3 threads Master script | 60 mins | t3.xlarge | 14 | 2xlarge DB |
7 | 6 threads Master script | 60 mins | t3.xlarge | 14 | 2xlarge DB |
8 | 8 threads Master script | 60 mins | t3.xlarge | 14 | 2xlarge DB |
9 | 8 threads Master script | 4 hr | t3.xlarge | 14 | xlarge DB |
10 | 3 threads Master script | 60 mins | t3.xlarge | 14 | 4xlarge DB |
Results and Comparisons
Performance dependency with increasing load with xlarge DB
Workflow | 2 threads response times | 3 threads response times | delta 2 vs 3 threads | 6 threads response times | delta 3 vs 6 threads | 8 threads response times | delta 6 vs 8 threads |
---|---|---|---|---|---|---|---|
AIE_TC: View Invoices Workflow | 7,191 | 6,991 | -0,2 | 11,612 | 4,621 | 14,218 | 2,606 |
BLS_TC: Blacklight simulation | 3,932 | 5,073 | 1,141 | 7,276 | 2,203 | 9,347 | 2,071 |
ETT_TC: Edit Bib RecordTagTable Group | 17,057 | 40,933 | 23,876 | 58,72 | 17,787 | 97,45 | 38,73 |
FYR_TC: Load Finance Ledgers Rollover | 0,39 | 0,522 | 0,132 | 0,536 | 0,014 | 0,773 | 0,237 |
ILR_TC: Create ILR | 1,482 | 2,414 | 0,932 | 2,446 | 0,032 | 3,072 | 0,626 |
ILR_TC: Get ItemId | 1,731 |