Overview

Tool/Framework

Current solution for acceptance/integration tests is based on Postman

Postman

• Tool Pros.:  
  • Well-known tool for most people.
  • Used in many projects as a tool for sharing REST collection for manual testing and development needs.
  • It is possible to run collections by CI/CD tools.
• Tool Cons.:
  
  • API test collection is stored as huge JSON file and as a result there are problems with tests reviewing and PR conflict merging.
  • There is no way to reuse utility methods from "Pre-request test"  across collections and as a result for reducing copy-paste all tests should be in one collection.
  • Data for the tests is distributed among several variable scopes (global, environment).
  • The tests are unreadable and not clear, if we use place-holders {{bodyForRequest}} in the body for PUT/POST. 
  • Develop asserts and prepare of the input data for the test cases is going on in a NotePad style without the possibility of the IDE.
  • There is no way to use Data Driven Test approach.
• Actual implementation problems:
  • Independent principle is violated from the F.I.R.S.T. : a lot test cases depend on the results of others.
  • One collection for one module.
  • Most tests duplicate API unit tests and one different that they use real DB.
  • Clean up of the system state is done by separate set of test, but sometimes it is not happen. As a result invalid data still in DB.
  • Utility methods from "Pre-request test", which do the same in different collections are named differently.
  • Placeholders are used for in the body for PUT/POST.
• Suggestions/Improvements
  • Functional quality checking by writing tests for features or for user flows.
  • Maybe make sense to place test collections under modules and not in the separate module.
  • Make all tests independent

There is an understanding that migration to other frameworks is practically excluded and will be quite a challenge, but it still makes sense to consider them.

REST-Assured

• Tool  Pros.:
  • BDD style test writing 
  • Can be combine with Cucumber : http://angiejones.tech/rest-assured-with-cucumber-using-bdd-for-web-services-automation
• Tool  Cons.:
• Actual implementation problems:
  • Used only for API unit tests
• Suggestions/Improvements

Cucumber

• Tool Pros.:
  • Popular in test automation world
  • BDD feature files used as definition for the test case steps
  • Support Data Driven Test technic
  • Easy to debug and start locally
  • Integrated with common CI/CD tools
  • Can be integrated with tools like https://rp.epam.com/ui/
  • Reports is generated in human readable style
  • Easy to create new test cases for new features
• Tool Cons.:
  • Need to implement engine for supporting DSL of the feature files
• Actual implementation problems:
• Suggestions/Improvements

Karate

• Tool Pros.:
  • Based on the popular Cucumber / Gherkin standard
  • Tests are super-readable - as scenario data can be expressed in-line, in human-friendly JSON, XML, Cucumber Scenario Outline tables, or a payload builder approach unique to Karate
  • Simpler and more powerful alternative to JSON-schema for validating payload structure and format - that even supports cross-field / domain validation logic
  • Scripts can call other scripts - which means that you can easily re-use and maintain authentication and 'set up' flows efficiently, across multiple tests
  • Embedded JavaScript engine that allows you to build a library of re-usable functions that suit your specific environment or organization
  • Built-in support for switching configuration across different environments (e.g. dev, QA, pre-prod)
  • Support for data-driven tests and being able to tag or group tests is built-in, no need to rely on an external framework
  • Standard Java / Maven project structure, and seamless integration into CI / CD pipelines - and support for JUnit 5
  • Option to use as a light-weight stand-alone executable - convenient for teams not comfortable with Java
  • Multi-threaded parallel execution, which is a huge time-saver, especially for integration and end-to-end tests
  • Comprehensive documentation
• Tool Cons.:
  • Need spent more time to learn DSL and script structure
  • Current version is 0.9.5
• Actual implementation problems:
• Suggestions/Improvements
  • Example project