SPIKE [FAT-1652]: mod-inn-reach: Study existing solutions for mocking (mock servers) and choose the appropriate one
Capabilities of interest
- deployment options (docker, command line etc)
- mock definition/configuration
- automatic mocking from provided API spec
- request verification
- stateful behaviour
- supporting of dynamic/conditional content
- simulating faults
WireMock
WireMock is an HTTP mock server. At its core it is web server that can be primed to serve canned responses to particular requests (stubbing) and that captures incoming requests so that they can be checked later (verification). All of WireMock’s features are accessible via its REST (JSON) interface and its Java API.
Deployment
a) Standalone process
The WireMock server can be run in its own process. Once the standalone JAR downloaded it can be run simply by doing this:
$ java -jar wiremock-jre8-standalone-2.33.1.jar
Various command line options available, can be listed with --help command. Some useful options:
--port
: Set the HTTP port number e.g. --port 9999. Use --port 0 to dynamically determine a port.
--verbose
: Turn on verbose logging to stdout
--root-dir
: Sets the root directory, under which mappings
and __files
reside. This defaults to the current directory.
b) Docker container
From version 2.31.0 WireMock has an official Docker image.
Start a single WireWock container with default configuration:
docker run -it --rm -p 8080:8080 --name wiremock wiremock/wiremock:2.33.1
The Docker image supports exactly the same set of command line arguments as the standalone version. These can be passed to the container by appending them to the end of the command e.g.:
docker run -it --rm -p 8443:8443 --name wiremock wiremock/wiremock:2.33.1 --https-port 8443 --verbose
Mock definition/configuration
a) Configuring via JSON over HTTP
A stub can be created by posting mapping to WireMock’s HTTP API:
$ curl -X POST \ --data '{ "request": { "url": "/get/this", "method": "GET" }, "response": { "status": 200, "body": "Here it is!\n" }}' \ http://localhost:8080/__admin/mappings/new
b) JSON file configuration
the JSON API can also be used via files. When the WireMock server starts it creates two directories under the current one: mappings
and __files
. JSON files containing multiple stub mappings can also be used.
c) Packaging the stubs into a standalone JAR
If you want to package your stubs into the standalone JAR, so you can distribute an executable JAR with all the stubs intact, you can do this using the --load-resources-from-classpath
option.
You could then run the packaged JAR as:
java -jar custom-wiremock.jar --load-resources-from-classpath 'wiremock-stuff'
Which will load your files and mappings from the packaged JAR.
Request verification
The WireMock server records all requests it receives in memory (at least until it is reset). This makes it possible to verify that a request matching a specific pattern was received, and also to fetch the requests’ details.
Verifying and querying requests relies on the request journal, which is an in-memory log of received requests.
Like stubbing, verification also uses WireMock’s Request Matching system to filter and query requests.
To verify that a request matching some criteria was received by WireMock at least once:
The criteria part in the parameter to postRequestedFor()
uses the same builder as for stubbing, so all of the same predicates are available. See Stubbing for more details.
To check for a precise number of requests matching the criteria, use this form:
Stateful behaviour
WireMock supports state via the notion of scenarios. A scenario is essentially a state machine whose states can be arbitrarily assigned. It starting state is always Scenario.STARTED
. Stub mappings can be configured to match on scenario state, such that stub A can be returned initially, then stub B once the next scenario state has been triggered.
Dynamic/conditional content
a) Transforming Responses
Via WireMock’s extension mechanism it is possible to dynamically modify responses, allowing header re-writing and templated responses amongst other things.
There are two ways to dynamically transform output from WireMock. WireMock stub mappings consist in part of a ResponseDefinition
. This is essentially a description that WireMock (sometimes) combines with other information when producing the final response.
ResponseDefinition
objects are “rendered” by WireMock into a Response
, and it is possible to interject either before or after this process when writing an extension, meaning you can either transform the ResponseDefinition
prior to rendering, or the rendered Response
afterwards.
A response transformer extension takes a Response
in its transform method’s parameter list and returns a Response
:
b) Response templating
Response headers and bodies, as well as proxy URLs, can optionally be rendered using Handlebars templates. This enables attributes of the request to be used in generating the response e.g. to pass the value of a request ID header as a response header or render an identifier from part of the URL in the response body.
When starting WireMock programmatically, response templating can be enabled by adding ResponseTemplateTransformer
as an extension e.g.
The boolean constructor parameter indicates whether the extension should be applied globally. If true, all stub mapping responses will be rendered as templates prior to being served.
Command line parameters can be used to enable templating when running WireMock standalone:
--global-response-templating
: Render all response definitions using Handlebars templates.
--local-response-templating
: Enable rendering of response definitions using Handlebars templates for specific stub mappings.
Parameter values can be passed to the transformer as shown below (or dynamically added to the parameters map programmatically in custom transformers):
These parameters can be referenced in template body content using the parameters.
prefix:
<h1>The MyCustomParameter value is {{parameters.MyCustomParameter}}</h1>
Simulating faults
In addition to being able to send back any HTTP response code indicating an error, WireMock is able to generate a few other types of problem.
a) Delayed responses
Per-stub fixed delays – A stub response can have a fixed delay attached to it, such that the response will not be returned until after the specified number of milliseconds
Global fixed stub delays – A fixed delay can be added to all stubs
Per-stub random delays – A delay can be sampled from a random distribution. This allows simulation of more specific downstream latencies, such as a long tail.
- Global random stub delays
Chunked Dribble Delay – In addition to fixed and random delays, response can be dribble back in chunks. This is useful for simulating a slow network and testing deterministic timeouts.
b) Bad responses
It is also possible to create several kinds of corrupted responses:
The Fault
enum has the following options: EMPTY_RESPONSE
, MALFORMED_RESPONSE_CHUNK
, RANDOM_DATA_THEN_CLOSE,
CONNECTION_RESET_BY_PEER
Karate Netty
Deployment
a) Standalone process
All of Karate (core API testing, parallel-runner / HTML reports, the debugger-UI, mocks and web / UI automation) is available as a single, executable JAR file. Look for the latest release on GitHub and scroll down to find the "Assets". And look for the file with the name: karate-<version>.jar
To start a mock server, the 2 mandatory arguments are the path of the feature file 'mocks' -m
and the port -p
java -jar karate.jar -m my-mock.feature -m my-2nd-mock.feature -p 8080
To directly start a mock from within a Karate test script use the karate.start()
API. The argument can be a string or JSON. If a string, it is processed as the path to the mock feature file, and behaves like the read()
function.
So starting a mock from a Karate test is simple. This example also shows how conditional logic can be used effectively.
Background: * def port = karate.env == 'mock' ? karate.start('cats-mock.feature').port : 8080 * url 'http://localhost:' + port + '/cats'
Mock definition/configuration
Writing a mock can get complicated for real-life API interactions, and most other frameworks attempt to solve this using declarative approaches, such as expecting you to create a large, complicated JSON to model all requests and responses. Karate's approach combines the best of both the worlds of declarative and imperative programming. Combined with the capability to maintain state in the form of JSON objects in memory, and Karate's native support for Json-Path, XML and embedded expressions
The Karate 'server' life-cycle is simple and has only 2 phases - the Background
and Scenario
.
Background
is executed on start-up. You can read files and set-up common functions and 'global' state here. Note that unlike the life-cycle of 'normal' Karate, the Background
is not executed before each Scenario
.
Here's an example of setting up a function to generate primary keys which can be invoked like this: uuid()
A server-side Feature
file can have multiple Scenario
sections in it. Each Scenario is expected to have a JavaScript expression as the content of the Scenario
description which we will refer to as the "request matcher".
On each incoming HTTP request, the Scenario
expressions are evaluated in order, starting from the first one within the Feature
. If the expression evaluates to true
, the body of the Scenario
is evaluated and the HTTP response is returned.
Request verification
Validate payloads if needed, using a simpler alternative to JSON schema:
Detailed description of schema validation is available at Schema Validation section of Karate framework.
Stateful behaviour
In-memory JSON and JsonPath solves for 'state' and filtering if needed:
Link to working example in git: demo-mock.feature
Dynamic/conditional content
Shaping the HTTP response is very easy - you just set a bunch of variables: response
,responseStatus
, esponseHeaders
With embedded-expressions, you can create dynamic responses with a minimum of effort:
Karate easily reads JSON or XML from files. So when you have large complex responses, you can do this:
Note that embedded expressions work even for content loaded using read()
Because of Karate's Java interop capabilities there is no limit to what can be done.
Simulating faults
Delayed responses can be defined by setting responseDelay
variable which accepts a value in milliseconds:
Summary
WireMock | Karate Netty | Comments | |
---|---|---|---|
Deployment / Standalone process | |||
Deployment / Docker | Кarate doesn't have official docker image, but an image can be easily created since there is a possibility to start standalone Java process | ||
Mock definition | |||
Request verification | The verification in WireMock can be done with standard matchers programmatically. Karate option is more intuitive as it allows to define something similar to JSON schemas. | ||
Stateful behaviour | WireMock is able to simulate stateful behaviour using so called scenarios but Karate makes it more natural with an ability to support real variables with saved states between the request calls | ||
Dynamic payload of responses | |||
Simulating faults | Karate has limited support for delayed responses but because of simplified integration with Java/JS any sort of customization can be easily added to response processing | ||
|
As the above tables states, both WireMock and Karate Netty provide the required functionality. But Karate Netty looks more suitable because the integration tests in Folio are already written with Karate and in general it has more advanced options for customization with direct support of Java/JS code inside Karate features (test scenarios).
So the winner is Karate Netty !
References
- WireMock documentation: https://wiremock.org/docs/
- Prism documentation: https://meta.stoplight.io/docs/prism/ZG9jOjYx-overview
- Karate Netty documentation: https://github.com/karatelabs/karate/tree/master/karate-netty
- BenTen project is a great example of the usage of Karate test-doubles: https://github.com/intuit/benten/blob/master/benten-mock/src/main/resources/benten-mock.feature
- Comparison of API simulation tools: https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools