Submitted Date	17 Mar 2021
Approved Date	25 Aug 2021
Status	ACCEPTED
Impact	LOW

Overrides/Supersedes

This decision was migrated from the Tech Leads Decision Log as part of a consolidation process. The original decision record can be found here.

RFC

N/A

Stakeholders

All Backend Developers

Contributors

Anton Emelianov (Deactivated)

Approvers

This decision was made by the Tech Leads group prior to the adoption of current decision making processes within the FOLIO project.

Background/Context

Some bugs can only be discovered by integrating modules together. It is challenging to identify these on an ongoing basis.Some integration tests were already implemented. They use Postman collections. They are simple to write but harder to maintain.It can be challenging to express more complex scenarios. Many of which were contributed by ThunderJet team.ThunderJet teams has considered a range of tools and recommend Karate tool. No alternative proposals has been submitted therefore Karate framework is preferred method for creating API integration tests in 2020.

Assumptions

N/A

Constraints

N/A

Rationale

API Integration Test Goal

Automated regression
Cross module interaction
Validate Producer/Consumer relationship
Permissions
No mock objects

QA Goals for 2021

Priority	Action Item	R1	R2	R3
1	Move testing activities to teams scratch environments	50%	100%
2	Build API Integration Test Coverage (Karate framework)	35%	50%	75%
3	Implement UI Integration Test (New BigTest) - 109 test cases	40%	100%
4	Implement manual sprint regression using testing community	20%	50%	100%
5	Implement UAT events using SIG resources	20%	50%	100%
6	PO to complete specs for performance acceptance test	100%
7	Implement performance acceptance test		50%	100%
8	Extend BugFest time frame (Approved)

Scope

Identify most important business flows
Add test case to cover a bug condition
Don't write tests for all possible conditions

Please note that the intent is not to replace rest assured/junit with karate but instead to get the benefits of both approaches

Create tests for new features

Review if new feature has important business workflows and include test creation for these workflows into estimates
Features are all ranked and it should determine if tests have to be created
SM need to drive NFR and make sure that Karate tests are scheduled into Sprints

Test plan creation and estimation

Before the end of R1 teams have to analyze existing API and identify important business flows and created stubs for them.
Create Jira tickets for implementation of this tests.
Create an estimate for each Jira ticket
Evaluate each workflow based on downstream dependencies

Blocked by:

FAT-145 - Annotate scenarios and customize Cucumber report for viewing "Skip" tests in results. CLOSED

FAT-146 - POC : Solution for mapping pure Gherkin scenarios with Karate feature runs OPEN

Coverage calculation, reporting

Coverage defined as % of implemented test cases to total test case count.
Coverage will be calculated and reported by Jenkins pipeline

Test execution before and after PR

Part of acceptance criteria - execute Katate tests before PR
Deploy module in Scratch env. and run applicable tests
Module changes and test changes are merged at the same time
Test will be run automatically after PR in Jenkins pipeline the next day:
1. https://jenkins-aws.indexdata.com/job/FOLIO_Reference_Builds/job/folio-api-tests-karate/
2. https://jenkins-aws.indexdata.com/job/FOLIO_Reference_Builds/job/folio-api-tests-postman/

These pipelines are currently broken - FOLIO-2982 - API tests environment (for karate, postman) is fail to spin up CLOSED

Changes to DOD

Every team should adjust DOD to require API Integration Tests

Deprecation of Postman tests

Mostly affects Spitfire and Thunderjet
Continue to use Postman tests if team doesn't have desire to switch
Passing tests are required by acceptance criteria
Postman test pipeline must be:
1. operational
2. have no failures
Defer decision when to deprecate these tests.

Decision

Accepted

Implications

Pros
- N/A
Cons
- N/A

Other Related Resources

Proposed Action Items

Each team should change definition of done to make API integration tests required
All new BE features must include Karate tests.
R1 Goal: Each team has to create a test plan for existing API interfaces
R1 Goal: Each team should provide estimates for the test plan. Estimates will be presented at the Capacity Planning meeting so that this work can be officially scheduled into sprints.

Meeting Notes

2020-04-29 Meeting notes	Karate Proposal
2020-05-27 Meeting notes	Karate Integration Tests PoC - Q&A
2020-08-05 Meeting notes	Implementation of Karate tests

GitHub repo and documentation: https://github.com/folio-org/folio-integration-tests
Karate resources are at the bottom of README.md
Karate API Integration Testing implementation tracking page

Browser not supported