Investigation of variability of test results

Overview

The test aims to investigate the variability level of performance test results for CI/CO actions.

https://folio-org.atlassian.net/browse/PERF-455

Summary

• Results show us that increasing amount of samples decreases the margin of error (MoE) and makes results more efficient, so a recommendation could be used following:

  • increase the number of virtual users for an increasing amount of samples for the same duration

  • increase the duration for a growing amount of samples

• The median of the test data set is more stable than the average value of the same data set

• Based on calculated results:

  • for CI/CO 8VU:

    • Check-in & Check-OUT AVG +- ~6.66% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs) 

    • Check-in Standard Deviation of the following metrics aggregations:

      • AVG - 6.53 ms

      • Median - 5.57 ms

      • 95pct - 11.31 ms

    • Check-out Standard Deviation of the following metrics aggregations:

      • AVG - 6.81 ms

      • Median - 8.11 ms

      • 95pct - 10.39 ms

  • for CI/CO 20 VU:

    • Check-in & Check-OUT AVG +- ~3.43% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)

    • Check-in Standard Deviation of the following metrics aggregations:

      • AVG - 8.07 ms

      • Median - 8.97 ms

      • 95pct - 9.79 ms

    • Check-out Standard Deviation of the following metrics aggregations:

      • AVG - 8.40 ms

      • Median - 8.70 ms

      • 95pct - 9.48 ms

Test config

Results

CI/CO 8 VU