You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 4
Next »
Overview
The test aims to investigate the variability level of performance test results for CI/CO actions.
PERF-455
-
Getting issue details...
STATUS
Summary
- Results show us that increasing amount of samples decreases the margin of error (MoE) and makes results more efficient, so a recommendation could be used following:
- increase the number of virtual users for an increasing amount of samples for the same duration
- increase the duration for an increasing amount of samples
- The median of the test data set is more stable than the average value of the same data set
- Based on calculated results:
- for CI/CO 8VU:
- Check-IN & Check-OUT AVG +- ~6.66% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)
- for CI/CO 20 VU:
- Check-IN & Check-OUT AVG +- ~3.43% (Confidence 95%, MoE for analysis could be 7% or even 10% AS NFRs/SLAs)
Test config
Test # | Test Conditions | Duration | Count | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | CI/CO 8 VU | 30 min | 10 | t3.medium | 1 |
2. | CI/CO 20 VU | 30 min | 10 | t3.medium | 1 |
Results
CI/CO 8 VU
|
|
|
|
|
|
| Response time |
|
| ID | Test | Job | Total | Request group | Samples | AVG, ms | Median, ms | 95pct, ms | Min, ms | Max, ms | Range, ms | StdDev, ms | MoE (95% CL) | CoV |
| 1 | CICO 8 VU | #2190 | 76885 | Check-IN | 910 | 492 | 479 | 594 | 396 | 2470 | 2069 | 94 | 6.11 | 19.11% |
| Check-OUT | 1273 | 768 | 751 | 913 | 629 | 2450 | 1821 | 104 | 5.71 | 13.54% |
| 2 | CICO 8 VU | #2191 | 76686 | Check-IN | 941 | 499 | 483 | 607 | 405 | 1410 | 1004 | 75 | 4.79 | 15.03% |
| Check-OUT | 1248 | 777 | 754 | 950 | 637 | 2150 | 1508 | 109 | 6.05 | 14.03% |
| 3 | CICO 8 VU | #2192 | 76559 | Check-IN | 925 | 500 | 481 | 611 | 406 | 2530 | 2124 | 97 | 6.25 | 19.40% |
| Check-OUT | 1255 | 778 | 757 | 944 | 640 | 2030 | 1393 | 104 | 5.75 | 13.37% |
| 4 | CICO 8 VU | #2193 | 75675 | Check-IN | 943 | 492 | 476 | 605 | 404 | 1480 | 1080 | 72 | 4.60 | 14.63% |
| Check-OUT | 1221 | 773 | 754 | 938 | 626 | 2070 | 1446 | 103 | 5.78 | 13.32% |
| 5 | CICO 8 VU | #2194 | 76810 | Check-IN | 916 | 486 | 468 | 586 | 395 | 1510 | 1112 | 79 | 5.12 | 16.26% |
| Check-OUT | 1267 | 768 | 743 | 940 | 626 | 2230 | 1602 | 121 | 6.66 | 15.76% |
| 6 | CICO 8 VU | #2195 | 77187 | Check-IN | 914 | 477 | 465 | 574 | 396 | 1190 | 791 | 57 | 3.70 | 11.95% |
| Check-OUT | 1278 | 756 | 730 | 922 | 621 | 2210 | 1590 | 107 | 5.87 | 14.15% |
| 7 | CICO 8 VU | #2196 | 76433 | Check-IN | 924 | 483 | 470 | 581 | 388 | 1400 | 1010 | 65 | 4.19 | 13.46% |
| Check-OUT | 1251 | 759 | 738 | 934 | 624 | 2190 | 1570 | 102 | 5.65 | 13.44% |
| 8 | CICO 8 VU | #2197 | 76460 | Check-IN | 926 | 490 | 479 | 586 | 396 | 1380 | 980 | 62 | 3.99 | 12.65% |
| Check-OUT | 1252 | 773 | 754 | 929 | 642 | 1980 | 1342 | 99 | 5.48 | 12.81% |
| 9 | CICO 8 VU | #2198 | 76134 | Check-IN | 951 | 490 | 477 | 590 | 399 | 1350 | 954 | 66 | 4.19 | 13.47% |
| Check-OUT | 1227 | 773 | 750 | 940 | 645 | 2100 | 1451 | 109 | 6.10 | 14.10% |
| 10 | CICO 8 VU | #2199 | 75919 | Check-IN | 901 | 488 | 475 | 591 | 404 | 1480 | 1080 | 65 | 4.24 | 13.32% |
| Check-OUT | 1255 | 768 | 745 | 929 | 635 | 1910 | 1278 | 104 | 5.75 | 13.54% |
where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE=z-score (95% CL=1.96) * StdDev/SQRT(Samples)
CoV - Coefficient of variance - calculated as CoV = StdDev / Mean
|
| AVG, ms | Median, ms | 95pct, ms | StdDev, ms |
| Check-IN | AVG | 489.70 | 475.30 | 592.50 | 73.20 |
| StdDev | 6.53 | 5.57 | 11.31 | 12.69 |
| CoV | 1.33% | 1.17% | 1.91% | 17.34% |
| Check-OUT | AVG | 769.30 | 747.60 | 933.90 | 106.20 |
| StdDev | 6.81 | 8.11 | 10.39 | 5.74 |
| CoV | 0.89% | 1.09% | 1.11% | 5.41% |
CI/CO 20 VU
|
|
|
|
|
|
| Response time |
|
| ID | Test | Job | Total | Request group | Samples | AVG, ms | Median, ms | 95pct, ms | Min, ms | Max, ms | Range, ms | StdDev, ms | MoE (95% CL) | CoV |
| 1 | CICO 20 VU | #2200 | 189419 | Check-IN | 2323 | 481 | 471 | 577 | 322 | 1480 | 1158 | 60 | 2.44 | 12.47% |
| Check-OUT | 3082 | 770 | 751 | 922 | 630 | 1840 | 1209 | 88 | 3.11 | 11.43% |
| 2 | CICO 20 VU | #2202 | 189504 | Check-IN | 2378 | 488 | 477 | 587 | 341 | 1560 | 1220 | 61 | 2.45 | 12.50% |
| Check-OUT | 3048 | 779 | 762 | 933 | 629 | 2140 | 1511 | 90 | 3.20 | 11.55% |
| 3 | CICO 20 VU | #2203 | 190315 | Check-IN | 2325 | 485 | 475 | 584 | 334 | 1460 | 1126 | 61 | 2.48 | 12.58% |
| Check-OUT | 3103 | 774 | 757 | 933 | 622 | 2000 | 1376 | 92 | 3.24 | 11.89% |
| 4 | CICO 20 VU | #2204 | 189181 | Check-IN | 2336 | 489 | 480 | 585 | 341 | 1470 | 1126 | 61 | 2.47 | 12.47% |
| Check-OUT | 3069 | 781 | 764 | 935 | 628 | 2140 | 1509 | 93 | 3.29 | 11.91% |
| 5 | CICO 20 VU | #2205 | 189659 | Check-IN | 2346 | 496 | 486 | 602 | 346 | 1420 | 1073 | 64 | 2.59 | 12.90% |
| Check-OUT | 3074 | 791 | 776 | 956 | 630 | 2060 | 1427 | 97 | 3.43 | 12.26% |
| 6 | CICO 20 VU | #2206 | 190052 | Check-IN | 2372 | 471 | 460 | 569 | 329 | 1520 | 1187 | 59 | 2.37 | 12.53% |
| Check-OUT | 3068 | 763 | 747 | 923 | 618 | 1910 | 1293 | 90 | 3.18 | 11.80% |
| 7 | CICO 20 VU | #2207 | 190360 | Check-IN | 2329 | 471 | 458 | 567 | 333 | 1460 | 1129 | 62 | 2.52 | 13.16% |
| Check-OUT | 3101 | 762 | 745 | 924 | 619 | 2000 | 1383 | 96 | 3.38 | 12.60% |
| 8 | CICO 20 VU | #2208 | 191772 | Check-IN | 2325 | 472 | 460 | 581 | 316 | 1460 | 1140 | 65 | 2.64 | 13.77% |
| Check-OUT | 3138 | 766 | 751 | 932 | 590 | 1980 | 1386 | 94 | 3.29 | 12.27% |
| 9 | CICO 20 VU | #2209 | 188207 | Check-IN | 2341 | 480 | 468 | 592 | 323 | 1480 | 1157 | 63 | 2.55 | 13.13% |
| Check-OUT | 3040 | 774 | 755 | 940 | 619 | 1830 | 1214 | 94 | 3.34 | 12.14% |
| 10 | CICO 20 VU | #2210 | 191878 | Check-IN | 2305 | 477 | 465 | 585 | 323 | 1100 | 773 | 62 | 2.53 | 13.00% |
| Check-OUT | 3151 | 772 | 754 | 937 | 605 | 2130 | 1522 | 98 | 3.42 | 12.69% |
where MoE (95% CL) - Margin of Error with 95% confidence level - calculated as MoE = z-score (95% CL=1.96) * StdDev/SQRT(Samples)
CoV - Coefficient of variance - calculated as CoV = StdDev / Mean
|
| AVG, ms | Median, ms | 95pct, ms | StdDev, ms |
| Check-IN | AVG | 481.00 | 470.00 | 582.90 | 61.80 |
| StdDev | 8.07 | 8.97 | 9.79 | 1.72 |
| CoV | 1.68% | 1.91% | 1.68% | 2.78% |
| Check-OUT | AVG | 773.20 | 756.20 | 933.50 | 93.20 |
| StdDev | 8.40 | 8.70 | 9.48 | 3.09 |
| CoV | 1.09% | 1.15% | 1.02% | 3.32% |
1 #2190 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676975322277&to=1676977229179&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=Check-In%20Controller
2 #2191 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676977805847&to=1676979750899&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
3 #2192 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676980200000&to=1676982180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
4 #2193 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676982300000&to=1676984400000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
5 #2194 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676984700000&to=1676986800000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
6 #2195 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676987700000&to=1676989680000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
7 #2196 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676990100000&to=1676992200000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
8 #2197 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676992800000&to=1676994900000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
9 #2198 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676995500000&to=1676997600000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
10 #2199 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676997900000&to=1677000000000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
11 #2200 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677000120000&to=1677002220000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
12 #2202 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677005160000&to=1677007260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
13 #2203 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677007500000&to=1677009480000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
14 #2204 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677009600000&to=1677011820000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
15 #2205 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677012000000&to=1677014100000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
16 #2206 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677049080000&to=1677051180000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
17 #2207 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677051600000&to=1677053580000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
18 #2208 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677054300000&to=1677056280000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
19 #2209 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677056820000&to=1677058980000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
20 #2210 http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1677059220000&to=1677061260000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All
Methodology/Approach
- Scripts used:
- DB Refresh - checkin-checkout-db-restore.sql
- DB Update - circ-data-load.sh
- Executed CI/CO with 8 and 20 concurrent vUsers for 30 minutes 10 times (for each, 20 in total). (On carrier-io used the artefact "circulation_checkInCheckOut_nolana.zip ")