Master Script stress test - NLA report

Owned by Roman_Fedynyshyn
Last updated: Aug 08, 2023 by Olga Kondratenko
11 min read

Overview

The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors. 

Summary

  • Few test sets was performed:
    • 2,3,6,8 threads tests with xlarge (default) DB;
    • 2,3,6,8 threads tests with 2xlarge DB;
    • test with longer (4 hr) duration for xlarge DB with 8 threads
    • 3 threads test with 4xlarge DB.
  • During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.
  • There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section
  • With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets
  • For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):
    1. ILR_TC: Get ItemId - 709%
    2. VIR_TC: inventory record monitoring - 579%
    3. VTT_TC: Inventory View Instance Source - 531%
    4. VRO_TC: View Vendor - 420%
    5. LO_TC: View Ledgers - 335%
    6. VB_TC: Inventory Choose Instance - 326%
    7. VH_TC: Inventory View Holdings - 322%
    8. VH_TC: Inventory Choose Holdings -308%
    9. ETT_TC: Edit Bib RecordTagTable Group - 304%
    10. VRO_TC: Edit Vendor - 289%


 * - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load

Recommendations & Jiras (Optional)

PERF-525 - Getting issue details... STATUS

  • It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

  • Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                         "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

    {
                        "name": "JAVA_OPTS",
                        "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
                    },

  • If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

{
    "name": "DB_MAXPOOLSIZE",
    "value": "500"
},
{
    "name": "DB_CONNECTION_TIMEOUT",
    "value": "60"
},

Test Runs 

Test #

Test Conditions

Duration 

Load generator size (recommended)Load generator Memory(GiB) (recommended)

Notes

(Optional)

1.

2 threads Master script60 minst3.xlarge14xlarge DB

2.

3 threads Master script60 minst3.xlarge14xlarge DB
36 threads Master script60 minst3.xlarge14xlarge DB
48 threads Master script60 minst3.xlarge14xlarge DB
52 threads Master script60 minst3.xlarge142xlarge DB
63 threads Master script60 minst3.xlarge142xlarge DB
76 threads Master script60 minst3.xlarge142xlarge DB
88 threads Master script60 minst3.xlarge142xlarge DB
98 threads Master script4 hrt3.xlarge14xlarge DB 
103 threads Master script60 minst3.xlarge144xlarge DB



Results and Comparisons

Performance dependency with increasing load with xlarge DB

Workflow 2 threads response times  3 threads response timesdelta 2 vs 3 threads   6 threads response times delta 3 vs 6 threads 8 threads response times  delta 6 vs 8 threads
AIE_TC: View Invoices Workflow7,1916,991-0,211,6124,62114,2182,606
BLS_TC: Blacklight simulation3,9325,0731,1417,2762,2039,3472,071
ETT_TC: Edit Bib RecordTagTable Group17,05740,93323,87658,7217,78797,4538,73
FYR_TC: Load Finance Ledgers Rollover0,390,5220,1320,5360,0140,7730,237
ILR_TC: Create ILR1,4822,4140,9322,4460,0323,0720,626
ILR_TC: Get ItemId1,7314,2322,5015,5741,3429,6984,124
ILR_TC: Get RequesterId0,0450,042-0,0030,0580,0160,0620,004
IO_TC: Create Invoices Workflow3,7453,7540,0095,6721,9187,3751,703
IO_TC: Delete Invoices Workflow3,1583,151-0,0075,22,0496,6821,482
IO_TC: Edit Invoices Workflow5,7055,7230,0189,6023,87912,2082,606
IO_TC: View Invoices Workflow2,532,540,014,071,535,0670,997
IRO_TC: Delete Item6,70215,3398,63719,6474,30832,79413,147
IRO_TC: Edit Item6,6915,5678,87718,7243,15729,13410,41
IRO_TC: View Item11,60231,37519,77339,998,61562,72522,735
LO_TC: Create Ledgers1,0351,023-0,0121,3810,3581,4760,095
LO_TC: Delete Ledgers0,4260,5030,0770,7850,2820,9740,189
LO_TC: Edit Ledgers0,430,4970,0670,7940,2970,9760,182
LO_TC: View Ledgers0,4310,4980,0670,780,2820,990,21
POO_TC Create Order3,0472,779-0,2683,6680,8894,3690,701
POO_TC Delete Order2,2522,182-0,073,0340,8524,4581,424
POO_TC Edit Order3,3753,365-0,014,5241,1596,4281,904
POO_TC View Order1,9872,030,0432,6580,6283,9151,257
PRO_TC: Create User2,2572,3560,0992,840,4843,0090,169
PRO_TC: Delete User1,2021,2960,0941,510,2141,7130,203
PRO_TC: Edit User2,4472,6130,1663,3160,7033,6640,348
PRO_TC: View User1,2071,2420,0351,5290,2871,690,161
PRV_TC: View Patron record Group0,1280,127-0,0010,160,0330,1840,024
VAR_TC: GoTo Marc Authority0,0840,1390,0550,123-0,0160,1540,031
VAR_TC: Search Marc Authority0,0860,08-0,0060,0890,0090,0960,007
VB_TC: Inventory Choose Instance3,5878,224,63311,9393,71919,3227,383
VB_TC: Inventory Search Instance0,0470,050,0030,0520,0020,0550,003
VB_TC: Load Inventory1,2382,4461,2082,108-0,3382,5770,469
VH_TC: Inventory Choose Holdings3,2187,6534,43510,7263,07318,1467,42
VH_TC: Inventory Search Holdings0,0260,02600,0290,0030,0320,003
VH_TC: Inventory View Holdings3,8469,7645,91814,3654,60124,42610,061
VH_TC: Load Inventory1,2962,5631,2672,141-0,4222,2080,067
VIH_TC: View instance holdings details6,05512,9436,88814,9682,02522,9047,936
VIR_TC: inventory record monitoring8,11213,1725,0622,7119,53933,06610,355
VRO_TC: Create Vendor4,8644,722-0,1427,9923,2710,4812,489
VRO_TC: Delete Vendor1,0490,958-0,0911,1110,1531,4340,323
VRO_TC: Edit Vendor28,65329,2760,62354,42725,15166,96412,537
VRO_TC: View Vendor3,9323,889-0,0435,631,7416,7481,118
VTT_TC: Inventory Choose Instance3,0877,2014,1149,9682,76716,2846,316
VTT_TC: Inventory Search Instance0,0420,0440,0020,0510,0070,0510
VTT_TC: Inventory View Instance Source2,4636,0243,5618,6422,61815,1416,499
VTT_TC: Load Inventory1,2352,4141,1792,188-0,2262,6070,419
Check-In Controller4,0128,0424,039,4281,38614,3134,885
Check-Out Controller5,3599,7194,3611,1191,415,8864,767

Performance dependency with increasing load with 2xlarge DB

Workflow 2 threads response times  3 threads response timesdelta 2 vs 3 threads   6 threads response times delta 3 vs 6 threads 8 threads response times  delta 6 vs 8 threads
AIE_TC: View Invoices Workflow3,4223,8960,4745,0661,176,61,534
BLS_TC: Blacklight simulation2,5833,1480,5653,9780,835,1511,173
ETT_TC: Edit Bib RecordTagTable Group22,1636,33814,17860,31923,98199,73439,415
FYR_TC: Load Finance Ledgers Rollover0,4010,5520,1510,5730,021
0
FYR_TC:FYR_Rollover Ledgers3126,7963347,388220,592----
ILR_TC: Create ILR1,6582,1150,4572,570,4553,3860,816
ILR_TC: Get ItemId2,2673,931,6636,4182,48811,4495,031
ILR_TC: Get RequesterId0,0460,0530,0070,0580,0050,0680,01
IO_TC: Create Invoices Workflow2,6082,7180,113,3890,6714,1690,78
IO_TC: Delete Invoices Workflow1,7281,9190,1912,5830,6643,1670,584
IO_TC: Edit Invoices Workflow3,4943,720,2264,9431,2236,1631,22
IO_TC: View Invoices Workflow1,5331,6310,0982,1740,5432,610,436
IRO_TC: Delete Item8,48713,1324,64520,9027,7734,15613,254
IRO_TC: Edit Item8,972145,02822,8028,80232,2269,424
IRO_TC: View Item15,54224,4318,88937,7313,29963,62125,891
LO_TC: Create Ledgers0,830,9110,0811,1460,2351,2210,075
LO_TC: Delete Ledgers0,1960,2460,050,370,1240,480,11
LO_TC: Edit Ledgers0,1910,2470,0560,3760,1290,4780,102
LO_TC: View Ledgers0,1950,2450,050,370,1250,4740,104
POO_TC Create Order2,7732,676-0,0973,1010,4253,2480,147
POO_TC Delete Order1,9942,1060,1122,5180,4122,6870,169
POO_TC Edit Order3,1353,114-0,0213,9350,8214,2330,298
POO_TC View Order1,7911,8350,0442,2290,3942,3050,076
PRO_TC: Create User2,1932,4490,2562,710,2613,1330,423
PRO_TC: Delete User1,1191,2550,1361,3940,1391,6320,238
PRO_TC: Edit User2,0732,2070,1342,5150,3082,9260,411
PRO_TC: View User1,1081,2630,1551,3640,1011,5940,23
PRV_TC: View Patron record Group0,0990,1160,0170,1380,0220,1640,026
VAR_TC: GoTo Marc Authority0,1210,1490,0280,1780,0290,2250,047
VAR_TC: Search Marc Authority0,10,099-0,0010,1170,0180,109-0,008
VB_TC: Inventory Choose Instance4,5767,042,46412,04521,3529,312
VB_TC: Inventory Search Instance0,0530,052-0,0010,060,0080,057-0,003
VB_TC: Load Inventory1,7162,2480,5322,6070,3593,380,773
VH_TC: Inventory Choose Holdings3,8496,6542,80510,8494,19519,0048,155
VH_TC: Inventory Search Holdings0,0320,03200,0380,0060,037-0,001
VH_TC: Inventory View Holdings5,0889,1444,05614,9935,84925,38910,396
VH_TC: Load Inventory1,8262,4070,5812,9070,53,8380,931
VIH_TC: View instance holdings details7,01310,7223,70916,5495,82726,94210,393
VIR_TC: inventory record monitoring5,1868,1222,93612,9794,85721,8128,833
VRO_TC: Create Vendor1,7342,0090,2752,7650,7563,2650,5
VRO_TC: Delete Vendor0,7180,790,0720,9560,1661,0430,087
VRO_TC: Edit Vendor8,0989,5791,48113,8714,29218,5684,697
VRO_TC: View Vendor1,2511,4230,1722,0660,6432,3680,302
VTT_TC: Inventory Choose Instance3,7866,0422,25610,0444,00217,297,246
VTT_TC: Inventory Search Instance0,0470,046-0,0010,0530,0070,052-0,001
VTT_TC: Inventory View Instance Source3,0225,3492,3278,9253,57615,9937,068
VTT_TC: Load Inventory1,7232,230,5072,5110,2813,2830,772
Check-In Controller4,8916,8451,95412,375,52517,7335,363
Check-Out Controller6,229,3413,12113,2843,94321,9648,68

Comparison xlarge tests with 2xlarge tests

Workflow 2 threads response times (xlarge)  2 threads response times 2xlargedelta  3 threads response times (xlarge)  3 threads response times 2xlargedelta 6 threads response times (xlarge) 6 threads response times (2xlarge)

delta


8 threads response times (xlarge)8 threads response times (2large)delta
AIE_TC: View Invoices Workflow7,1913,422-3,7696,9913,896-3,09511,6125,066-6,54614,2186,6-7,618
BLS_TC: Blacklight simulation3,9322,583-1,3495,0733,148-1,9257,2763,978-3,2989,3475,151-4,196
ETT_TC: Edit Bib RecordTagTable Group17,05722,165,10340,93336,338-4,59558,7260,3191,59997,4599,7342,284
FYR_TC: Load Finance Ledgers Rollover0,390,4010,0110,5220,5520,030,5360,5730,0370,7730,7840,011
FYR_TC:FYR_Rollover Ledgers -3126,7963126,796 -3347,3883347,388 - --

0
ILR_TC: Create ILR1,4821,6580,1762,4142,115-0,2992,4462,570,1243,0723,3860,314
ILR_TC: Get ItemId1,7312,2670,5364,2323,93-0,3025,5746,4180,8449,69811,4491,751
ILR_TC: Get RequesterId0,0450,0460,0010,0420,0530,0110,0580,05800,0620,0680,006
IO_TC: Create Invoices Workflow3,7452,608-1,1373,7542,718-1,0365,6723,389-2,2837,3754,169-3,206
IO_TC: Delete Invoices Workflow3,1581,728-1,433,1511,919-1,2325,22,583-2,6176,6823,167-3,515
IO_TC: Edit Invoices Workflow5,7053,494-2,2115,7233,72-2,0039,6024,943-4,65912,2086,163-6,045
IO_TC: View Invoices Workflow2,531,533-0,9972,541,631-0,9094,072,174-1,8965,0672,61-2,457
IRO_TC: Delete Item6,7028,4871,78515,33913,132-2,20719,64720,9021,25532,79434,1561,362
IRO_TC: Edit Item6,698,9722,28215,56714-1,56718,72422,8024,07829,13432,2263,092
IRO_TC: View Item11,60215,5423,9431,37524,431-6,94439,9937,73-2,2662,72563,6210,896
LO_TC: Create Ledgers1,0350,83-0,2051,0230,911-0,1121,3811,146-0,2351,4761,221-0,255
LO_TC: Delete Ledgers0,4260,196-0,230,5030,246-0,2570,7850,37-0,4150,9740,48-0,494
LO_TC: Edit Ledgers0,430,191-0,2390,4970,247-0,250,7940,376-0,4180,9760,478-0,498
LO_TC: View Ledgers0,4310,195-0,2360,4980,245-0,2530,780,37-0,410,990,474-0,516
POO_TC Create Order3,0472,773-0,2742,7792,676-0,1033,6683,101-0,5674,3693,248-1,121
POO_TC Delete Order2,2521,994-0,2582,1822,106-0,0763,0342,518-0,5164,4582,687-1,771
POO_TC Edit Order3,3753,135-0,243,3653,114-0,2514,5243,935-0,5896,4284,233-2,195
POO_TC View Order1,9871,791-0,1962,031,835-0,1952,6582,229-0,4293,9152,305-1,61
PRO_TC: Create User2,2572,193-0,0642,3562,4490,0932,842,71-0,133,0093,1330,124
PRO_TC: Delete User1,2021,119-0,0831,2961,255-0,0411,511,394-0,1161,7131,632-0,081
PRO_TC: Edit User2,4472,073-0,3742,6132,207-0,4063,3162,515-0,8013,6642,926-0,738
PRO_TC: View User1,2071,108-0,0991,2421,2630,0211,5291,364-0,1651,691,594-0,096
PRV_TC: View Patron record Group0,1280,099-0,0290,1270,116-0,0110,160,138-0,0220,1840,164-0,02
VAR_TC: GoTo Marc Authority0,0840,1210,0370,1390,1490,010,1230,1780,0550,1540,2250,071
VAR_TC: Search Marc Authority0,0860,10,0140,080,0990,0190,0890,1170,0280,0960,1090,013
VB_TC: Inventory Choose Instance3,5874,5760,9898,227,04-1,1811,93912,040,10119,32221,3522,03
VB_TC: Inventory Search Instance0,0470,0530,0060,050,0520,0020,0520,060,0080,0550,0570,002
VB_TC: Load Inventory1,2381,7160,4782,4462,248-0,1982,1082,6070,4992,5773,380,803
VH_TC: Inventory Choose Holdings3,2183,8490,6317,6536,654-0,99910,72610,8490,12318,14619,0040,858
VH_TC: Inventory Search Holdings0,0260,0320,0060,0260,0320,0060,0290,0380,0090,0320,0370,005
VH_TC: Inventory View Holdings3,8465,0881,2429,7649,144-0,6214,36514,9930,62824,42625,3890,963
VH_TC: Load Inventory1,2961,8260,532,5632,407-0,1562,1412,9070,7662,2083,8381,63
VIH_TC: View instance holdings details6,0557,0130,95812,94310,722-2,22114,96816,5491,58122,90426,9424,038
VIR_TC: inventory record monitoring8,1125,186-2,92613,1728,122-5,0522,71112,979-9,73233,06621,812-11,254
VRO_TC: Create Vendor4,8641,734-3,134,7222,009-2,7137,9922,765-5,22710,4813,265-7,216
VRO_TC: Delete Vendor1,0490,718-0,3310,9580,79-0,1681,1110,956-0,1551,4341,043-0,391
VRO_TC: Edit Vendor28,6538,098-20,55529,2769,579-19,69754,42713,871-40,55666,96418,568-48,396
VRO_TC: View Vendor3,9321,251-2,6813,8891,423-2,4665,632,066-3,5646,7482,368-4,38
VTT_TC: Inventory Choose Instance3,0873,7860,6997,2016,042-1,1599,96810,0440,07616,28417,291,006
VTT_TC: Inventory Search Instance0,0420,0470,0050,0440,0460,0020,0510,0530,0020,0510,0520,001
VTT_TC: Inventory View Instance Source2,4633,0220,5596,0245,349-0,6758,6428,9250,28315,14115,9930,852
VTT_TC: Load Inventory1,2351,7230,4882,4142,23-0,1842,1882,5110,3232,6073,2830,676
Check-In Controller4,0124,8910,8798,0426,845-1,1979,42812,372,94214,31317,7333,42
Check-Out Controller5,3596,220,8619,7199,341-0,37811,11913,2842,16515,88621,9646,078


Comparison to separate workflows stress tests run

 * - with xlarge DB

CICO: Check-In/Check-Out

Workflow

 2 threads response times

master script

2 threads response times separate workflowdelta

  3 threads response times

master script

3 threads response times separate workflowdelta

   6 threads response times

master script

6 threads response times separate workflow delta

 8 threads response times

master script

 8 threads response times separate workflow  delta 
CICO_TC: Check-In Controller4,0120.653-3.3598,0420.610-7.4329,4280.594-8.83414,3130.580-13.733
CICO_TC: Check-Out Controller5,3591.124-4.2359,7191.069-8.65011,1191.034-10.08515,8861.022-14.864

IRO: Item record operation

Workflow

 2 threads response times

master script

2 threads response times separate workflowdelta

  3 threads response times

master script

3 threads response times separate workflowdelta

   6 threads response times

master script

6 threads response times separate workflow delta

 8 threads response times

master script

 8 threads response times separate workflow  delta 
IRO_TC: Delete Item6,7020.916-5.78615,3390.987-14.35219,6470.760-18.88732,7940.790-32.004
IRO_TC: Edit Item6,690.923-5.76715,5670.990-14.57718,7240.764-17.96029,1340.787-28.347
IRO_TC: View Item11,6021.218-10.38431,3751.321-30.05439,991.035-38.95562,7251.063-61.662

mod-inventory and mod-inventory-storage are the most loaded by this workflow.

ETT: Edit Bib RecordTagTable

Workflow

 2 threads response times

master script

2 threads response times separate workflowdelta

  3 threads response times

master script

3 threads response times separate workflowdelta

   6 threads response times

master script

6 threads response times separate workflow delta

 8 threads response times

master script

 8 threads response times separate workflow  delta 
ETT_TC: Edit Bib RecordTagTable Group17,0572.201-14.85640,9331.996-38.93758,722.441-56.27997,453.091-94.359

mod-quick-marc and mod-inventory-storage are the most loaded by this workflow.

Xlarge DB stress test

Memory Utilization

No memory leaks were defined. All of a modules has jumps in memory however those are not significant.

One drop of memory usage can be seen on a screen - it's mod-source-record-manager and mod-source-record-storage. this drop happened due to redeploy to newer versions. 


CPU Utilization 



most loaded modules: 

  • nginx-okapi - spikes up to 373% 
  • mod-configuration - spikes up to 272%
  • mod-users - spikes to 250%
  • mod-inventory-storage up to 85%
  • okapi - up to 85%
  • mod-finance-storage up to 85%
  • mod-finance up to 75%



RDS CPU Utilization 




2Xlarge DB stress test


Memory Utilization


CPU Utilization 


most loaded modules: 

  • nginx-okapi - spikes up to 500% 
  • mod-users - spikes to 450%
  • mod-configuration - spikes up to 180%
  • mod-finance-storage up to 150%
  • mod-finance up to 120%
  • mod-inventory-storage up to 97%
  • okapi - up to 90%


RDS CPU Utilization 


4 Hours stress test (xlarge DB)

During previous two testing sets we've found that RDS CPU reached ±100% almost from the beginning of a test. So the purpose of this test is to check if system can handle such high load during longer duration without breaking of DB or other components of a system.

This time DB CPU usage reached 100% from the beginning of a test and stays at this point up to the end. System didn't fail and was stable after test.

Memory Utilization


CPU Utilization 


most loaded modules: 

  • nginx-okapi - spikes up to 401% 
  • mod-users - spikes to 230%
  • mod-configuration - spikes up to 120%
  • mod-finance-storage up to 60%
  • mod-finance up to 60%
  • mod-inventory-storage up to 88%
  • okapi - up to 77%


RDS CPU Utilization 



4Xlarge DB stress test


Memory Utilization



CPU Utilization 



most loaded modules: 

  • nginx-okapi - spikes up to 401% 
  • mod-users - spikes to 326%
  • mod-configuration - spikes up to 116%
  • mod-finance-storage up to 165%
  • mod-finance up to 111%
  • mod-inventory-storage up to 78%
  • okapi - up to 77%


RDS CPU Utilization 




Database analysis

AAS  or an active session is a database connection that has submitted a request to the database but has not yet received the response. Measuring the average number of active concurrent sessions over time provides a clear picture of the load on the database. https://aws.amazon.com/blogs/database/analyzing-amazon-rds-database-workload-with-performance-insights/




Slow queries: 

a lot of (this queries taking more than 1-2 seconds to execute)

SELECT COUNT(*) FROM fs00001069_mod_inventory_storage Open question - which workflow is calling that taking 

  • SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.instance (AVG latency up to 2359.19 ms 0.03 calls/sec)
  • SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.holdings_record (AVG latency up to 1465.36 ms 0.03 calls/sec)
  • SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.item (AVG latency up to 1580.26 ms 0.03 calls/sec)

SELECT fs00001069_mod_organizations_storage.count_estimate(?) (AVG latency up to 261.67 ms 0.20 calls/sec)


long one

2023-07-06 23:19:28 UTC:10.23.10.189(47150):fs00001069_mod_finance_storage@folio:[16764]:LOG: duration: 1808067.854 ms statement: DO
$$
begin
PERFORM fs00001069_mod_finance_storage.budget_encumbrances_rollover('
{ "id": "f2803540-7d7c-4b2a-8068-40dcd0725d16", "ledgerId": "cb8ba034-a78c-46f6-b41c-c0faec101c39", "rolloverType": "Preview", "fromFiscalYearId": "2284fa2a-926c-49bb-b949-0a791fd8353b", "toFiscalYearId": "bad9ec7d-186c-48c4-a64d-a23676044eab", "restrictEncumbrance": true, "restrictExpenditures": true, "needCloseBudgets": true, "currencyFactor": 2, "budgetsRollover": [ { "rolloverAllocation": true, "rolloverBudgetValue": "Available", "setAllowances": false, "adjustAllocation": 0, "addAvailableTo": "Available" } ], "encumbrancesRollover": [ { "orderType": "Ongoing", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "Ongoing-Subscription", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "One-time", "basedOn": "InitialAmount", "increaseBy": 0 } ], "metadata": { "createdDate": "2023-07-06T22:49:20.492+00:00", "createdByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce", "updatedDate": "2023-07-06T22:49:20.492+00:00", "updatedByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce" } }');
end;
$$ LANGUAGE plpgsql;






Appendix

Infrastructure

PTF -environment nptf 

  • 8 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 instance of db.r6.xlarge,db.r6.2xlarge,db.r6.4xlarge database (depending on a test)
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
  • mod-source-record-storage DB_CONNECTION_TIMEOUT =40;
  • mod-source-record-manager DB_RECONNECTATTEMPTS = 3, DB_RECONNECTINTERVAL=1000


Backend:

Module
nptf-pvt
Wed Jul 12 11:47:40 UTC 2023		Task Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-remote-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.224920447210243960512512false
mod-finance-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.4.12102489612870088128false
mod-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.13.12102489612876888128false
mod-agreements1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:5.5.22309625801282048384512false
mod-ebsconet1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.0.0212481024128700128256false
edge-sip21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.0.02102489612876888128false
mod-organizations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.7.02102489612870088128false
mod-settings1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.021024896200000false
mod-data-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:2.7.11204818442561292384512false
edge-dematic1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.0.021024896128000false
mod-search1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:2.0.122592248040014405121024false
mod-tags1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.0.12102489612876888128false
mod-authtoken1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.13.021440115251292288128false
mod-inventory-update1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.0.12102489612876888128false
mod-notify1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.0.02102489612876888128false
mod-configuration1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.12102489612876888128false
mod-orders-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.5.02102489651270088128false
edge-caiasoft1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.0.021024896128000false
mod-login-saml1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.6.12102489612876888128false
mod-erm-usage-harvester1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.3.02102489612876888128false
mod-gobi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.6.02102489612870088128false
mod-licenses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:4.3.12248023121281792384512false
mod-password-validator1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.0.0214401298128768384512false
mod-bulk-operations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:1.0.523072260010241536384512false
mod-graphql1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.11.02102489612876888128false
mod-finance1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.7.12102489612870088128false
mod-erm-usage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.5.22102489612876888128false
mod-copycat1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.4.02102489612876888128false
mod-permissions2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.3.12168415445121024384512false
mod-entities-links1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:1.0.2225922480400144001024false
pub-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2022.03.022102489612876800false
mod-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.6.622048144010241024384512false
edge-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:4.11.02102489625676888128false
edge-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.8.12102489612876888128false
mod-users-bl1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.5.021440115251292288128false
mod-inventory-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.022208195210241440384512false
mod-invoice1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.6.221440115251292288128false
edge-ea-data-export1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ea-data-export:3.9.02102489612876888128false
mod-user-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.7.22102489612876888128false
mod-sender1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.10.02102489612876888128false
edge-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.6.121512136010241440384512false
mod-data-export-worker1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.0.1223072260010242048384512false
mod-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.5.02102489612876888128false
mod-circulation-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.02153614401024896384512false
mod-source-record-storage5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.624096368810243076384512false
mod-calendar1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:2.4.22102489612876888128false
mod-event-config1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.5.02102489612876888128false
mod-courses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.72102489612876888128false
mod-inventory3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.422880259210241814384512false
mod-email1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.15.32102489612876888128false
mod-circulation1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.4228802592153676888128false
mod-pubsub1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.9.12153614401024922384512false
mod-di-converter-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.0.22102489612876888128false
edge-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:2.8.12102489612870088128false
edge-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.6.02102489612876888128false
mod-template-engine1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.18.02102489612876888128false
mod-users1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.1.12102489612876888128false
mod-patron-blocks1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.8.021024896102476888128false
mod-audit1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.7.02102489612876888128false
mod-source-record-manager5579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.324096368810243076384512false
nginx-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2022.03.0221024896128000false
mod-quick-marc1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.01228821761281664384512false
nginx-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2022.03.0221024896128000false
okapi-b2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.13168414401024922384512false
mod-feesfines1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:18.2.12102489612876888128false
mod-invoice-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.6.02102489612870088128false
mod-service-interaction1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:2.2.22204818442561290384512false
mod-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:5.5.22102489612876888128false
mod-data-export1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:4.7.111024896102476888128false
mod-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.11.322248200010241440384512false
edge-connexion1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.0.62102489612876888128false
mod-notes1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.0.121024896128322128128false
mod-kb-ebsco-java1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:3.13.02102489612876888128false
mod-data-export-spring1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:2.0.11204818442561292200256false
mod-login1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.9.02144012981024768384512false
mod-organizations-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.5.12102489612870088128false
pub-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2022.03.022102489612876800false
mod-data-import-converter-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import-converter-storage:1.15.22102489612876888128false
mod-eusage-reports1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:1.3.02102489612876888128false
 Changes of configurations
Module
nptf-pvt		Task Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-configuration2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.12102489651276888128false
mod-inventory-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.022208195215361440384512false
mod-source-record-storage6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.624096368815363076384512false
mod-users2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.1.12102489651276888128false
mod-source-record-manager6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.324096368815363076384512false
mod-quick-marc2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.02228821765121664384512false
nginx-okapi2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2022.03.02210248961024000false
okapi-b4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.13168414401536922384512false
mod-circulation-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.02153614401536896384512false
mod-circulation2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.422880259215362048384512false

Different configurations testing

Summary

  • General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

  • Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                         "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

    {
                    "name": "JAVA_OPTS",
                    "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
                },
  • If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Getting issue details... STATUS :
                {
                    "name": "DB_MAXPOOLSIZE",
                    "value": "500"
                },
                {
                    "name": "DB_CONNECTION_TIMEOUT",
                    "value": "60"
                },



1# Normal load test
WorkflowNO DINO DI 2xlarge dbComparison of difference between DB size with the same service changes (%)NO DI Updated 2xlarge db
mod-users,
mod-circulation,
mod-inventory-storage,
mod-inventory,
nginx-okapi,
mod-srm,
mod-srs		NO DI Updated
mod-users,
mod-circulation,
mod-inventory-storage,
mod-inventory,
nginx-okapi,
mod-srm,
mod-srs		Comparison of difference between DB size with the same service configurations (%)
AIE_TC: View Invoices Workflow3.322.46325.812.542.7618
BLS_TC: Blacklight simulation1.5260.96636.691.0611.1164.92
CICO_TC_Check-In Controller1.2470.97821.571.0811.1173.22
CICO_TC_Check-Out Controller2.0371.62220.371.7021.7754.11
ETT_TC: Edit Bib RecordTagTable Group2.5342.4812.092.8412.137-32.94
ILR_TC: Create ILR0.6690.50923.910.5430.586.37
ILR_TC: Get ItemId0.1060.07628.30.0860.078-10.25
ILR_TC: Get RequesterId0.0280.02510.710.0280.03212.5
IO_TC: Create Invoices Workflow2.0831.61622.411.6811.7272.66
IO_TC: Delete Invoices Workflow1.4051.13619.141.1411.2518.79
IO_TC: Edit Invoices Workflow2.72.10522.032.1522.3136.96
IO_TC: View Invoices Workflow1.1060.93415.551.0721-7.2
IRO_TC: Delete Item1.1830.90223.750.9390.9854.67
IRO_TC: Edit Item1.3160.89731.830.9771.0022.49
IRO_TC: View Item1.3220.99224.961.1521.1630.94
LO_TC: Create Ledgers0.7220.57220.770.6150.6210.96
LO_TC: Delete Ledgers0.060.04328.330.0580.0626.45
LO_TC: Edit Ledgers0.0820.088-7.310.0580.046-26.08
LO_TC: View Ledgers0.0530.0424.520.0440.0464.34
POO_TC Create Order2.0381.57622.661.7061.7854.42
POO_TC Delete Order1.0490.9766.951.0640.84-26.66
POO_TC Edit Order1.6331.4213.041.0541.28217.78
POO_TC View Order1.0240.85616.40.7990.8586.87
PRO_TC: Create User1.491.03630.461.081.22611.9
PRO_TC: Delete User0.8130.65918.940.6310.73914.61
PRO_TC: Edit User1.3121.15811.731.1061.093-1.18
PRO_TC: View User0.7550.51132.310.6870.597-15.07
PRV_TC: View Patron record Group0.0970.06236.080.0840.072-16.66
ULR_TC: Users loan Renewal Transaction1.711.3620.461.4311.5095.16
VAR_TC: GoTo Marc Authority0.0610.083-36.060.10.073-36.98
VAR_TC: Search Marc Authority0.1270.136-7.080.1630.144-13.19
VB_TC: Inventory Choose Instance0.4490.31529.840.3450.3634.95
VB_TC: Inventory Search Instance0.0630.066-4.760.0840.052-61.53
VB_TC: Load Inventory0.5260.43816.730.4460.4684.7
VH_TC: Inventory Choose Holdings0.4340.31128.340.340.3585.02
VH_TC: Inventory Search Holdings0.0460.067-45.650.10.048-108.33
VH_TC: Inventory View Holdings0.4490.33425.610.3570.3611.1
VH_TC: Load Inventory0.5320.4711.650.4870.514.5
VIH_TC: View instance holdings details1.7181.40518.211.5291.494-2.34
VIR_TC: inventory record monitoring0.9460.77318.280.7720.8276.65
VRO_TC: Create Vendor1.5270.96536.81.0051.13811.68
VRO_TC: Delete Vendor0.5650.43123.710.4240.47410.54
VRO_TC: Edit Vendor7.2974.64736.314.4225.80423.81
VRO_TC: View Vendor0.8860.67523.810.6970.87920.7
VTT_TC: Inventory Choose Instance0.4380.28634.70.3220.3374.45
VTT_TC: Inventory Search Instance0.0380.04-5.260.0530.037-43.24
VTT_TC: Inventory View Instance Source0.1740.11832.180.1270.1377.29
Total response time52.240.222.9841.743.84.79
2# Stress test load 2 threads
WorkflowBefore services adjustment with DIAfter adjustments with DINO DINO DI Updated
DB 2x large DB CPU usage
decreased from 99.4 to 86.9 %		Comparison of difference between DB size with the same service changes (%)

NO DI Updated DB 2x large
mod-users,
mod-circulation,
mod-inventory-storage,
mod-inventory,
nginx-okapi,
mod-srm,
mod-srs

AIE_TC: View Invoices Workflow12.40910.9367.1344.13742.014.302
BLS_TC: Blacklight simulation5.7424.6843.3211.79146.071.976
Check-In Controller4.5043.3942.2992.546-10.742.621
Check-Out Controller5.9265.4723.2813.794-15.633.669
ETT_TC: Edit Bib RecordTagTable Group19.90545.34710.2411.7-14.2511.801
ILR_TC: Create ILR1.8111.4680.9921.029-3.721.105
ILR_TC: Get ItemId1.7181.4140.8580.947-10.370.997
ILR_TC: Get RequesterId0.0910.0710.050.06-200.044
IO_TC: Create Invoices Workflow6.1335.743.7252.88622.522.854
IO_TC: Delete Invoices Workflow5.1814.8043.1682.0236.231.968
IO_TC: Edit Invoices Workflow9.8728.3185.94.00432.133.912
IO_TC: View Invoices Workflow4.2693.8212.5331.77230.041.73
IRO_TC: Delete Item6.9366.5513.7324.598-23.24.164
IRO_TC: Edit Item7.0926.4983.5314.072-15.324.013
IRO_TC: View Item10.0347.614.7816.36-33.026.039
LO_TC: Create Ledgers1.6151.3290.9260.9012.690.898
LO_TC: Delete Ledgers0.5090.3940.2330.11650.210.12
LO_TC: Edit Ledgers0.5130.3950.2310.1248.050.132
LO_TC: View Ledgers0.5180.3730.2310.1152.380.119
POO_TC Create Order5.8774.7222.9413.231-9.863.295
POO_TC Delete Order4.0553.5332.071.946.281.915
POO_TC Edit Order6.0754.932.4712.563-3.722.071
POO_TC View Order3.8643.1031.8671.65511.351.545
PRO_TC: Create User3.6752.9221.9932.198-10.281.938
PRO_TC: Delete User1.9391.6921.0621.161-9.321.062
PRO_TC: Edit User3.9693.4312.1752.0227.031.853
PRO_TC: View User1.9691.6851.1161.139-2.061.049
PRV_TC: View Patron record Group0.2410.1940.1280.1178.590.103
ULR_TC: Users loan Renewal Transaction4.8354.9693.1332.22129.12.292
VAR_TC: Click Authorized Record0.2510.1840.1140.167-46.490.155
VAR_TC: GoTo Marc Authority0.0730.0680.0660.093-40.90.081
VAR_TC: Search Marc Authority0.1070.090.0810.136-67.90.102
VB_TC: Inventory Choose Instance3.7763.4821.9132.149-12.332.11
VB_TC: Inventory Search Instance0.050.0490.0460.074-60.860.058
VB_TC: Load Inventory1.331.180.8771.067-21.661.01
VH_TC: Inventory Choose Holdings3.4873.0981.7722.168-22.342.098
VH_TC: Inventory Search Holdings0.0260.0260.0280.048-71.420.035
VH_TC: Inventory View Holdings3.7973.1591.8952.495-31.662.522
VH_TC: Load Inventory1.3951.1920.8631.024-18.650.977
VIH_TC: View instance holdings details7.6926.6093.8333.6893.753.66
VIR_TC: inventory record monitoring10.9859.3596.4322.93254.413.238
VRO_TC: Create Vendor8.8567.4584.7351.91359.592.056
VRO_TC: Delete Vendor2.7282.2011.0860.82224.30.917
VRO_TC: Edit Vendor44.22836.8828.4518.42470.399.557
VRO_TC: View Vendor7.045.2774.6111.37170.261.332
VTT_TC: Inventory Choose Instance3.2552.7691.6631.888-13.521.896
VTT_TC: Inventory Search Instance0.0460.0470.0440.063-43.180.056
VTT_TC: Inventory View Instance Source2.371.841.0981.347-22.671.323
VTT_TC: Load Inventory1.3561.2210.8971.073-19.621.036

Total response time



136.6104.123.79%103.8

Methodology/Approach

As a base to generate stress load we've used MasterScript with excluded controllers on transactions per minute. So in this case threads is not equal to Virtual users as threads will generate more load than users.