Mission

Having a central and consistent environment will be key to display and convey test results to all teams. Equally important is having a common testing methodology and interpretation of test results that will enable everyone in the community to understand the results the same way.  

The Performance Task Force (PTF) mission is to support the FOLIO community by

• providing a performance testing environment
• executing performance tests to establish baseline performance metrics, to check for performance regressions, and to investigate performance issues. 
• creating and maintaining standards for performance test scripts.

Operation

PTF 

• Set up carrier-io framework, performance testing environment, maintaining environment
• Develop performance testing guidelines and JMeter standards
• Execute tests and share results with teams, and create JIRA defects or stories for teams to address performance issues.
• Investigate performance issues reported by the community
• Review test cases contributed by teams
• Balance short term versus long term goals
  • Short Term 
    • Release performance system before the end of Q2 2020
    • Work directly with early implementers, e.g., Chalmers, to get their feedback on the FOLIO's performance
    • Create and execute tests for the highest/most impactful scenarios.
    • Investigate performance issues
    • Receive test scripts from teams and help them execute tests
    • Communicate with teams to share results and analysis
  • Long Term
    • Hooking carrier-io framework into CI pipelines.
    • Evolve carrier-io framework/system into a package that can be distributed across teams to enable teams to execute tests on their own.
    • Train teams to use carrier-io so they can execute the tests themselves
• Scope of analysis:
• FOLIO and integration points
• Do not analyze third-party systems, which include FOLIO clients, but may make recommendations of how to better integrate with FOLIO
• Do not fix performance issues, only identify and report them

Product Owners

• Work with their team to identify performance scenarios to be investigated. Create JIRA tickets on the PTF board.
• Help PTF to triage and and prioritize PTF tests backlog.  Stephanie Buck will facilitate the discussions between PTF and the POs on priority.
• If performance issues are reported by PTF product owners would prioritize them on their team's board to have the issues fixed.

Teams

• Contribute JMeter test scripts to be run
• Keep JMeter test scripts up to date.
• Contribute seed data that are relevant to the test scenarios that the test scripts would run against. 
• Contribute scripts to load and unload the seed data
• Fix performance issues found by PTF

Environment

PTF will use the open source carrier-io framework in the performance testing environment. The framework consists of a set of tools (Grafana, InfluxDB, JMeter, Telegraph) that are packaged to quickly execute tests and monitor results. Tests will be invoked from a common stand-alone Jenkins job that either could execute various test cases on demand, or can be integrated into a CI pipeline that executes tests at a configurable frequency, on check-in, for example. Initially it is desirable to start out with a stand-alone Jenkins job that is simple to set up and use. FOLIO in this environment will be refreshed every two weeks, at the end (or beginning) of a sprint after all codes have passed their definition of done. This is done to ensure that the environment is stable and that tests can be run without functional issues.

Communication

Being able to share and communicate results with teams to get them on the same page is crucial to the mission of PTF.  To achieve this goal PTF will initially set out to directly inform teams of results and in the future once the team’s processes are matured PTF will hold weekly meetings to share results with the product owners and wider community.