Deal with technically failed Test Run
If a Test Configuration was executed, there are two reasons the Test Run can fail:
- A functional error like a failed assert or validation feedback.
- A technical error (Exception) in the Application.
An example of a functional error: Value was different from the Assert value
An example of a technical error: Execution user does not have access to a certain resource
This document covers the second reason; for Asserts, please read the reference guide.
The icons on a Teststep showing the different possible outcomes are as follows:
|A successful test||Functionally failed||Technically failed|
An important difference between a functional error and a technical error during a Test Run, is that when MTA encounters a functional error, it will continue the Test Run; when MTA encounters a technical error, it will abort the Test Run.
If a Test Run is aborted, it may sometimes be difficult to deal with the resulting data, that is usually incomplete. Currently, all the data that was committed in a Microflow or Persisted in a Teststep by MTA, is permanently stored in the database, except for the last running Test Case where the technical error occured. All the data in the last running Test Case is rolled back.
To clarify, consider the following Test Suite example:
- [Test Case] Create Customer masterdata
- [Test Case] Create Order for Customer
- [Test Case] Fill Order and create Invoice
- [Test Case] Send Invoice to Customer
- [Test Case] Cleanup Order and Customer data
Note that this (Process test) example does not follow the Best Practice of cleaning up data before the test execution.
Now let's execute this Test Suite:
- Create Customer masterdata
- Create Order for Customer
- Fill Order and create Invoice
- Send Invoice to Customer
- Cleanup Order and Customer data
The first four Test Cases were executed, but the fourth failed, leading to a rollback for that Test Case. This means that all the data from the first three Test Cases still exists because the last Test Case was not executed. Also, without inspecting the Teststeps in the fourth Test Case, it is not clear if any external source like an API or SMTP server was contacted, or that the Test Case failed before encountering the respective Teststep.
The best way to avoid this scenario is to follow the Best Practice of cleaning up data before the test execution, and to use custom (Microflow) logic where data is first checked and then deleted, instead of using Retrieve and Delete Object Teststeps for this.
Missing anything? Let us know!
Last updated 13 may 2022