Overview

Carrier-io performance testing platform consists of a Jenkins job where users can configure options for their test and kicks off the test.  This page documents various options for configuring a performance test.

The Jenkins job is at https://jenkins-aws.indexdata.com/job/performance_framework/job/carrier-io-single-test.

Parameters

testType

Test Type is a way to categorize the tests using one of the options of fix_load, capacity, baseline, or longevity.  The test run's JMeter data can be retrieved in the JMeter Performance Grafana dashboard by selecting the test type and test name. For example:

Options

• fix_load: Fixed load tests are designed to run at a fixed load, usually by the number of virtual users. These are ad-hoc tests and have no special meaning other than running at a particular load.
• capacity: Capacity tests are designed to probe and understand the system's capability given at particular loads.  Usually it takes a few test runs at various loads to determine the system's capacity.  For example, there is a need to see how much load a system with x RAM and y CPUs can handle, a set of capacity tests would be run to determine the maximum load, the load at which performance starts to degrade, the load at which performance is optimal. These tests are tagged as "capacity".
• baseline: Baseline tests are tests to establish a baseline performance of a system. Usually these tests involve one 1 user or however a baseline is defined by the workflow and application.  The result of this test is used to compare with the result of another baseline test, usually of a different version, to see if performance has improved or regressed.
• longevity: Longevity tests are run for at least 8 hours, normally 12-24 hours, to detect abnormal patterns over a long period of time. Usually memory leaks or rising CPU utilizations are discovered by longevity tests. It is good to determine if the system behaves consistently under a steady load, mimicking production environments.

testName

TestName is the name of the JMeter Test Plan name. 

bucket

Bucket is a carrier-io concept. It is like a directory with a customizable logical name to hold test artifacts.  PTF creates different buckets for different apps or different sets of related apps in FOLIO.  A "circulation" bucket, for example, contains test artifacts of check-in and check-out apps.  When specifying on Jenkins it must be the name of the bucket that holds the test artifact with which the test is run.

targetUrl

TargetURL is the URL of Okapi against which the test is executed. This value does not have the https:// portion.

influxDbUrl

InfluxDbUrl is the URL of the InfluxDB instance where JMeter test metrics and related test data are stored. There is an InfluxDB instance in each FOLIO team's area on Rancher. This is the URL of this InfluxDB instance. The test data will be sent to this InfluxDB for storage.  In PTF's case it's ptf-influxdb.ci.folio.org

influxDbPort

Port of InfluxDB instance in each FOLIO team's Rancher area, usually 8086.

users

Number of concurrent virtual users to execute the test. The more number of users the greater the load that is sent to the API endpoints. PTF has been using 1 (for baseline), 5, 8, and 20 users.  8 users is about the average number of concurrent users at most library at any one time. FOLIO Performance usually starts to degrade or performing poorly with 20 users.

rampUp

RampUp is the total number of seconds it takes to add virtual users. If there are 8 users, and rampUp time is set at 80, it means that on average a virtual user is introduced into the system every 10 seconds.  If we want the users to be added every 20 seconds, then the rampUp time is 160 seconds. How long between adding a virtual user depends on the area of FOLIO being tested. We certainly don't want to overwhelm the system by blasting it with too much load initially.

duration

Duration of the test, in seconds. A typical fixed-load test takes about 30 minutes, or 1800s seconds.  A longevity test may take 24 hours, or 86,400 seconds.

loadGeneratorsCount

Number of load generators. Currently only 1 load generator suffices because there is no need to test with over 20 users. But in other situations when warranted, a higher number of users, in the hundreds range, may need multiple load generators. 

artifact*

Artifact is the name of the zipped up test package (.zip) deposited in a carrier-io's bucket (see "bucket" above).  An artifact consists of the JMeter test script along with any supporting files, such as authentication configuration or data that is expected by the test when it runs, such as instanceIDs or user barcodes, etc... Typically artifact is a zipped folder with a modified authentication configuration file zipped up and given a logical name for the test.

reportingInstanceUrl*

The URL of the carrier-io instance to which test data will be forwarded to. This value does not have the https:// portion.

distribution*

An optional value denoting the distribution of load in a complex workflow test, one that has at least two flows.  For example, a check-in-check-out test that has two workflows may have a load distribution value of 40-60. The first value, 40, is for check-in, while the second value, 60, is for check out.  The order of the numbers correspond to the order of what the test expects.  If a test has multiple flows, then the values are separated by multiple hyphens and again, the order of the values corresponds to what the test expects.  The values should add up to 100.

targetRegion*

AWS region where the FOLIO deployment (i.e., Okapi) is hosted. Use default value.

targetCluster*

Full name of the AWS ECS cluster, the infrastructure that hosts the FOLIO deployment.  Use default value.

targetEnv*

Name of the AWS account (prod, int, or devqa) which hosts the FOLIO deployment. Use default value.

instanceType*, spotPrice*

These two parameters describe the Spot ec2 instance that is automatically spun up when the Jenkins job executes.  This on-demand ec2 instance contains the load generator and is created and torn down dynamically to save costs. The instanceType is the AWS instance type, typically t3.medium, and the spotPrice is the price of the spot instance, usually $0.05 per hour. Use default values.

*Parameters in a slightly older version of carrier-io, are not applicable in the FOLIO community's Jenkins job.