Skip to main content
added 2 characters in body
Source Link
Robbie Dee
  • 9.8k
  • 2
  • 26
  • 54

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.
The

The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.
The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.

The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.

Source Link

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.
The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.