Investigation of variability of test results

Owned by Artem Vronik
Last updated: Feb 27, 2023
2 min read

Overview

The test aims to investigate the variability level of performance test results for CI/CO actions.

PERF-455 - Getting issue details... STATUS

Summary

  • Results show us that increasing amount of samples decreases the margin of error (MoE) and makes results more efficient, so a recommendation could be used following:
    • increase the number of virtual users for an increasing amount of samples for the same duration
    • increase the duration for a growing amount of samples
  • The median of the test data set is more stable than the average value of the same data set
  • Based on calculated results:
    • for CI/CO 8VU:
      • Check-in & Check-OUT AVG +- ~6.66% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs) 
      • Check-in Standard Deviation of the following metrics aggregations:
        • AVG - 6.53 ms
        • Median - 5.57 ms
        • 95pct - 11.31 ms
      • Check-out Standard Deviation of the following metrics aggregations:
        • AVG - 6.81 ms
        • Median - 8.11 ms
        • 95pct - 10.39 ms
    • for CI/CO 20 VU:
      • Check-in & Check-OUT AVG +- ~3.43% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)
      • Check-in Standard Deviation of the following metrics aggregations:
        • AVG - 8.07 ms
        • Median - 8.97 ms
        • 95pct - 9.79 ms
      • Check-out Standard Deviation of the following metrics aggregations:
        • AVG - 8.40 ms
        • Median - 8.70 ms
        • 95pct - 9.48 ms

Test config

Test #

Test Conditions

Duration 

CountLoad generator size (recommended)Load generator Memory(GiB) (recommended)

1.

CI/CO 8 VU30 min10t3.medium1

2.

CI/CO 20 VU30 min10t3.medium1

Results

CI/CO 8 VU








Response time
IDTestJobTotalRequest groupSamplesAVG, msMedian, ms95pct, msMin, msMax, msRange, msStdDev, msMoE (95% CL)CoV
1CICO 8 VU#219076885Check-in9104924795943962470206994                6.1119.11%
Check-OUT127376875191362924501821104                5.7113.54%
2CICO 8 VU#219176686Check-IN9414994836074051410100475                4.7915.03%
Check-OUT124877775495063721501508109                6.0514.03%
3CICO 8 VU#219276559Check-IN9255004816114062530212497                6.2519.40%
Check-OUT125577875794464020301393104                5.7513.37%
4CICO 8 VU#219375675Check-IN9434924766054041480108072                4.6014.63%
Check-OUT122177375493862620701446103                5.7813.32%
5CICO 8 VU#219476810Check-IN9164864685863951510111279                5.1216.26%
Check-OUT126776874394062622301602121                6.6615.76%
6CICO 8 VU#219577187Check-IN914477465574396119079157                3.7011.95%
Check-OUT127875673092262122101590107                5.8714.15%
7CICO 8 VU#219676433Check-IN9244834705813881400101065                4.1913.46%
Check-OUT125175973893462421901570102                5.6513.44%
8CICO 8 VU#219776460Check-IN926490479586396138098062                3.9912.65%
Check-OUT12527737549296421980134299                5.4812.81%
9CICO 8 VU#219876134Check-IN951490477590399135095466                4.1913.47%
Check-OUT122777375094064521001451109                6.1014.10%
10CICO 8 VU#219975919Check-IN9014884755914041480108065                4.2413.32%
Check-OUT125576874592963519101278104                5.7513.54%

where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE=z-score (95% CL=1.96) * StdDev/SQRT(Samples)

CoV - Coefficient of variance - calculated as CoV = StdDev / Mean



AVG, msMedian, ms95pct, msStdDev, ms
Check-inAVG  489.70        475.30    592.50         73.20

StdDev      6.53            5.57      11.31         12.69

CoV1.33%1.17%1.91%17.34%
Check-OUTAVG  769.30        747.60    933.90       106.20

StdDev      6.81            8.11      10.39           5.74

CoV0.89%1.09%1.11%5.41%

 

CI/CO 20 VU








Response time
IDTestJobTotalRequest groupSamplesAVG, msMedian, ms95pct, msMin, msMax, msRange, msStdDev, msMoE (95% CL)CoV
1CICO 20 VU#2200189419Check-in23234814715773221480115860                2.4412.47%
Check-OUT30827707519226301840120988                3.1111.43%
2CICO 20 VU#2202189504Check-IN23784884775873411560122061                2.4512.50%
Check-OUT30487797629336292140151190                3.2011.55%
3CICO 20 VU#2203190315Check-IN23254854755843341460112661                2.4812.58%
Check-OUT31037747579336222000137692                3.2411.89%
4CICO 20 VU#2204189181Check-IN23364894805853411470112661                2.4712.47%
Check-OUT30697817649356282140150993                3.2911.91%
5CICO 20 VU#2205189659Check-IN23464964866023461420107364                2.5912.90%
Check-OUT30747917769566302060142797                3.4312.26%
6CICO 20 VU#2206190052Check-IN23724714605693291520118759                2.3712.53%
Check-OUT30687637479236181910129390                3.1811.80%
7CICO 20 VU#2207190360Check-IN23294714585673331460112962                2.5213.16%
Check-OUT31017627459246192000138396                3.3812.60%
8CICO 20 VU#2208191772Check-IN23254724605813161460114065                2.6413.77%
Check-OUT31387667519325901980138694                3.2912.27%
9CICO 20 VU#2209188207Check-IN23414804685923231480115763                2.5513.13%
Check-OUT30407747559406191830121494                3.3412.14%
10CICO 20 VU#2210191878Check-IN2305477465585323110077362                2.5313.00%
Check-OUT31517727549376052130152298                3.4212.69%

where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE = z-score (95% CL=1.96) * StdDev/SQRT(Samples)

CoV - Coefficient of variance - calculated as CoV = StdDev / Mean



AVG, msMedian, ms95pct, msStdDev, ms
Check-inAVG  481.00        470.00    582.90         61.80

StdDev      8.07            8.97         9.79           1.72

CoV1.68%1.91%1.68%2.78%
Check-OUTAVG  773.20        756.20    933.50         93.20

StdDev      8.40            8.70         9.48           3.09

CoV1.09%1.15%1.02%3.32%

 

1 #2190 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676975322277&to=1676977229179&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=Check-In%20Controller

2 #2191 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676977805847&to=1676979750899&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

3 #2192 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676980200000&to=1676982180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

4 #2193 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676982300000&to=1676984400000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

5 #2194 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676984700000&to=1676986800000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

6 #2195 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676987700000&to=1676989680000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

7 #2196 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676990100000&to=1676992200000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

8 #2197 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676992800000&to=1676994900000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

9 #2198 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676995500000&to=1676997600000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

10 #2199 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676997900000&to=1677000000000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All


11 #2200 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677000120000&to=1677002220000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

12 #2202 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677005160000&to=1677007260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

13 #2203 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677007500000&to=1677009480000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

14 #2204 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677009600000&to=1677011820000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

15 #2205 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677012000000&to=1677014100000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

16 #2206 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677049080000&to=1677051180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

17 #2207 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677051600000&to=1677053580000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

18 #2208 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677054300000&to=1677056280000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

19 #2209 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677056820000&to=1677058980000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

20 #2210 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677059220000&to=1677061260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Methodology/Approach

  1. Scripts used:
    1. DB Refresh - checkin-checkout-db-restore.sql 
    2. DB Update - circ-data-load.sh
  2. Execute CI/CO with 8 and 20 concurrent vUsers for 30 minutes 10 times (for each, 20 times in total).  (On carrier-io used the artefact "circulation_checkInCheckOut_nolana.zip ")
  3. Gather all needed metrics and their aggregations via the corresponding Grafana dashboard 'JMeter Performance - test Copy' 
  4. Calculate the Standard Deviation of the following metrics aggregations: AVG, Median, and 95pct