Karate tests and TestRail integration
The recent changes to the Karate core has introduced the ability to run the Karate tests and sync their results with the Test Rail portal. Below we will look how to prepare your test classes to enable result syncing in your workflows.
Test grouping
The first step would be to segregate your Test Rail-based tests into dedicated Java classes that contain a test group in the name (e.g. Smoke, Extended and Critical Path), for example CrossModulesCriticalPathApiTest has CriticalPath in the name and will allow Maven to filter classes by this pattern. For more information about test groups please consult this guide, but in short we don’t want to sync all results with the Test Rail, but only those Java classes that hold Karate features created explicitly from Test Rail cases.
Add TestRailClient
When you have Java classes arranged, the next step would be to add a new instance of TestRailClient to the overloaded constructor of TestIntegrationService.
This will configure TestIntegrationService to listen for specific env vars and prepare necessary infrastructure, such the HTTP client as well to pull/cache all case ids for the given test run.
Tag scenarios
When the classes are fully configured, you can start creating your Karate features from the Test Rail cases. To mark your feature scenario for syncing, you must tag them with the appropriate case id. For example, if your case number is C451636, then the Karate feature must have the same added as a tag, e.g. @C451636 above the scenario.
This must be done for each scenario in the Karate feature that implements a test case. If the tag is incorrectly formatted, does not exist in the test run or is missing, the results for this scenario will not be synced.
Run with env vars
Finally, after all is configured and scenarios are tagged, we can run our tests either in the IntelliJ, by passing the correct env vars or from the CI.
Name | Description |
TESTRAIL_HOST | The URL pointing to the Test Rail Portal |
TESTRAIL_USERNAME | User on whose behalf results are created |
TESTRAIL_PASSWORD | User password |
TESTRAIL_RUN_ID | The id of test run that house our current tests |
Consult Team Thunderjet for the necessary credentials or if you need a dedicated user for your needs, reach our to our Lead QA for help.
Run test in IntelliJ:
Run tests in the CI as a job:
Choose the correct branch, necessary submodules and test groups.
Next, set the project id and the run id. The run id can be retrieved from the URL.
Check the results
The results can be checked on the Test Rail after the completion of your Java classes. The integration creates them after the tenant is unentitled and deleted for each class.
If your scenario is passing, the result with be created with status id 1 and if it has failed it will be saved with status id 5, together with a comment.