Overview

To date the PTF has been executing JMeter tests manually. It takes a lot of time to rerun regression tests and can only test FOLIO at the end of the release cycle during Bugfest. Often time final testing results would come in after Bugfest and bug fixing is complete, which is too late to incorporate the performance feedback into the release. The PTF needs an automated test pipeline that can run regression performance tests frequently during development so that teams will be able to address performance issues in time for the release. This document will collect ideas and present a roadmap of how to build such an automated pipeline.

Goals

The goal is to have an automated test pipeline that regularly runs a suite of regression test scripts and compare the results against the last few runs.

• Four kinds of tests need to be run: individual workflows, multiple workflows, multi-tenant workflows, and a 24-hour longevity test. Individual workflows tests are needed to be able to clearly see the performance degradation of just that workflow without it being affected by other workflows.

• Each test will run 15 minutes to get consistent data. The first five minutes may have irregular spikes to ramp up time and warm up time of the modules so the results are not reliable. The remaining 10 minutes should establish a consistent load and performance from the system.

• Tests should be run on a large dataset, Bugfest-like or larger.

• Tests should be able to run repeatedly. Either the environment will need to be torn down and rebuilt each time a test is run, or there would be database scripts to restore dataset’s state.

• Tests should run at least twice a week, and may take advantage of the weekend to run longevity regression tests

Test Cases

In addition to the existing test cases (and scripts) that the PTF already has, new test cases will added from the this spreadsheet. Some core workflows identified are listed here

Infrastructure

• Two Jenkins pipelines needed:

  1. A Jenkins pipeline to create the FOLIO snapshot environment. Pipeline and environment hosted and maintained by FSE.

     • The FOLIO env is going to be destroyed manually by the PTF (via some Jenkins job) in order to give PTF time to troubleshoot and analyze performance

     • If the next test run is scheduled while the env is still alive from the previous test run, then this next test run should be skipped.

     • Be able to rerun tests manually and to run modified tests manually on demand. This means that the env should be able to be resurrected outside of scheduled run times.

     • The environment is a snapshot environment; it should pull the latest code once a week and upgrade all the modules

       • If the code has a data migration script, it is expected that the environment runs the migration script once upon spinning up.

     • Environment should be HA, similar to a typical FSE’s deployment. It should have an expected number of Eureka modules, including Keycloak, kong, etc.. Ideally it should also have the PTF recommended modules and sidecars settings.

     • The environment uses the selc database which has a consortia, including a central tenant and 5 member tenants. In the future a few more standalone tenants will be added to run non-ECS tests.

     • If it is decided that TLS should be enabled in production going forward, then this env will need to have TLS enabled as well.

     • For troubleshooting:

       • The environment and pipeline should already export logs and metrics into S3 buckets or Grafana already (and maybe into PTF’s “space”)

       • For rerunning tests, we could allocate huge numbers of items and user barcodes, for example, to continue running tests with them rather than needing to recreate the env or refresh the database. Alternatively, the database snapshot may be restored and swapped in. Or we can just recreate the env altogether.

  2. A Jenkins pipeline to run JMeter tests scripts from a Github repository (maybe current PTF’s perf-testing repo but from a different folder that has curated test scripts). Test environment to be hosted and maintained by Kitfox on Rancher.

     1. Pipeline needs to run PTF’s DMS service

     2. Tests are run daily, with each day running a different set of tests.

        1. Monday: individual workflows ECS

        2. Tuesday: Individual workflows non-ECS

        3. Wednesday: Multiple workflows single tenant ECs and non-ECS

        4. Thursday: Individual workflows on multiple tenants ECS and non-ECS

        5. Friday: Multiple workflows on multiple tenants ECS and non-ECS

        6. Saturday: Longevity

     3. JMeter test results are exported into a HMTL page report, which is to be housed in the Report Portal. The report should contain the typical results like total of samples, average, min, max, percentile response times. The report should also compare the results against the previous two test runs to identify trends. Also comparing against the last release’s results as a baseline. Report should also have an SLA column to compare current results against.

        1. A page to summarize the test results of all the workflows and have links to the individual workflows' reports

        2. Each workflow will have its own report page, information in this page includes:

           1. response times

           2. module versions

           3. comparisons against previous test runs

           4. more TBD and to be added in phase 2

     4. Reports should be saved for at least one month.

     5. Color-coded comparisons for each workflow

        • Green: Very good results +/- 5% from SLA

        • Yellow: 5-10% from SLA

        • Red: 15% from SLA

Dataset and Test Data

Starting out the dataset for the environment will be the PTF’s version of LoC’s dataset, selc dataset, which has only LoC’s inventory data. This dataset’s size is good for volume testing.

Each test needs to have the expected data to run with. For the first iteration, it may be simpler to focus on just getting the necessary data created for the tests directly needs in the LoC perf dataset. This may require new data creation scripts to generate such data. For example, CICO requires the user barcodes and item barcodes. These barcodes should already exist in the database and a simple script is needed to pull them from the database.

(Beyond the first iteration, new data generation scripts could be created to generate the necessary test data on top of any dataset for a test. These new scripts for the CICO workflow will include generating at the very least: items, users, patron groups, service points, loan policies, loan rules, other inventory reference data. The graph of data dependencies may be deep so the scripts will need to have a limit as to what dependent data to generate.)

Tests and Test Scripts

The following tests will be run each time:

• Individual workflows

• Multi-workflows

• Multi-workflows-multi-tenants

• Longevity using multi-workflows script will be run once a week

• ECS for all the sets of tests listed above

• non-ECS for all the sets of tests listed above

Individual workflows: JMeter test scripts for individual workflows will be employed. Some may need to be reviewed and updated to mimic the new release’s behaviors. Other new workflow test scripts will be created.

Multi-workflows: There exists a master script that contains multiple workflows. This script will need to be maintained and run as well.

Multi-workflows/multi-tenants: There exists a master script that contains multiple workflows that multiple tenants can execute. This script will need to be maintained and run as well.