Performance Testing

Owned by Kateryna Senchenko
Last updated: Aug 31, 2018

FOLIO-1406 - Getting issue details... STATUS


Jenkins pipeline project to test Folio performance is located in the repo - folio-perf-test.

JMeter is used as a tool for performance testing. Each JMeter script corresponds to an individual API in a module. Basic scenario is to create new data by POST request and DELETE it once test completes.

Example workflow used to test the mod-inventory-storage item-storage API:

1. GET /item-storage/items
2. POST /item-storage/items
Body Data:
{ "holdingsRecordId": "2b92dc88-d44d-4dc3-bd9e-89596adbe54b", "materialTypeId": "1a54b431-2e4f-452d-9cae-9cee66c9a892", "permanentLoanTypeId": "2b94c631-fca9-4892-a730-03ee529ffe27" }
3. GET /item-storage/items/5ee11d91-f7e8-481d-b079-65d708582ccc
4. PUT /item-storage/items/5ee11d91-f7e8-481d-b079-65d708582ccc
{ "id" : "1e399da3-782b-4312-9e3b-ef7ee2238a81", "holdingsRecordId": "d031681a-dbb9-49d0-9367-90419168d0c0", "pieceIdentifiers": ["This is test identifier"], "notes": ["This is test notes"], "materialTypeId": "1a54b431-2e4f-452d-9cae-9cee66c9a892", "permanentLoanTypeId": "2b94c631-fca9-4892-a730-03ee529ffe27" }
5. GET /item-storage/items/5ee11d91-f7e8-481d-b079-65d708582ccc
6. DELETE /item-storage/items/5ee11d91-f7e8-481d-b079-65d708582ccc
7. GET /item-storage/items/5ee11d91-f7e8-481d-b079-65d708582ccc



Tests are scheduled to run in Jenkins. Jenkins job runs nightly and triggers about 16000 http requests. New environment is created every time the tests run and is torn down once testing is complete. 


Stages followed by the Jenkins pipeline:

  1. Check out.
  2. Create environment - call AWS CloudFormation template to create three EC2 instances. One m5.large instance for Okapi, and two m5.xlarge instances for Modules and Database.
  3. Check environment -  wait for all instances to be fully ready.
  4. Bootstrap DB - pull and start PostgreSQL Docker image on DB EC2 instance.
  5. Bootstrap Okapi - pull and start Okapi Docker image used by snapshot-stable site from Docker hub and start it on Okapi EC2 instance.
  6. Bootstrap Modules - pull and start all modules Docker images used by snapshot-stable site.
  7. Populate data - pull EBSCO sample data from a public S3 bucket to populate database. Sample data is a Harvard dataset which could be considered similar to production version.
  8. Run JMeter tests.
  9. Tear down environment - delete the AWS stack.


EC2 instances characteristics:


Reports are generated in Jenkins using JMeter Performance plugin.

Following metrics are collected in the Performance Report:

  • Average response times (ART) for each transaction
  • Min and Max response times
  • Median
  • Failure rate and errors/warnings in the logs
  • Throughput

SLA Goals:

  • The average response time (AVG RT) for the JMeter captured transaction should not be more than 1000 milliseconds.
  • The percent of CPU utilization on any module should not be more than 50%.
  • JMeter tests running nightly in Jenkins pipeline as a job will fail if even a single test fails.


Example performance report: