Support for Exception-Creation-Strategies: AssertionFailedException, TestAbortedException, ...

[mmichaelis]

JUnit 4 knew 3 exit states:

• Failure - triggered by failed assertions
• Skipped - triggered by failed assumptions
• Error - triggered by any other exception

Based on that we created a framework for system tests which:

1. Provide a function which retrieves a value from the system under test.
2. Embed this function into a Wait-object
3. To trigger waiting the Wait-object has methods like:
   • assertThat(Matcher)
   • assumeThat(Matcher)
   • waitUntil(Matcher)

Internally the process was always the same: Probe the system again and again and compare the retrieved value via the function with the matcher. After continuous failures until a given timeout the exception was raised: either an AssertionError, AssumptionViolatedException or a RuntimeException (more specifically WaitTimeoutException).

So only at one central point the decision was made which exception to throw. This was easy, as all exceptions used the same constructors.

This possibility will break with the current state of the AssertionFailedException.

Is this a valid use-case worth to be supported? If yes, we need to provide some solution.

The only problem to have some base exception for all types is the AssertionError in the hierarchy as already noted in issue #4.

If issue #4 is fixed we could also introduce another base class ComparisonFailedException.

Problem then: The ComparisonFailedException does not really fit into the hierarchy of TestSkippedException and especially TestAbortedException. Most of the time TestSkippedException and TestAbortedException won't know any expected and actual. Nevertheless this is what I want to signal, like:

• the test skipped, because the test expected Firefox as Browser but it was Chrome
• the test is in error because the system under test did not reach the state to do the actual assertion

If you think this idea is worth to give it a try I will see, if I find a solution - but the solution would also take the idea of #4 into account as this would ease to find an appropriate solution.

[mmichaelis]

For a more concrete example have a look at: https://github.com/CoreMedia/joala and more
specifically to:

• joala/DefaultCondition.java at master · CoreMedia/joala

The Condition is an object which knows how to retrieve values from a system under test, it knows how to determine if the value matches the expectations and it knows how long to wait for the system under test to reach the expected value. To start a condition to trigger the last step is to choose the strategy it will fail with, which is to use {{assertThat}}, {{assumeThat}} or {{waitUntil}}.

The sketch in here might make it clearer:

• Package javadoc net.joala.condition

[marcphilipp]

JUnit 4 knew 3 exit states:

• Failure - triggered by failed assertions
• Skipped - triggered by failed assumptions
• Error - triggered by any other exception

JUnit 4 actually doesn't distinguish between errors and failures. I believe that was a JUnit 3 feature. However, some IDEs (e.g. Eclipse) display AssertionErrors different from other exceptions.

So only at one central point the decision was made which exception to throw. This was easy, as all exceptions used the same constructors.

Please correct me if I'm wrong but don't they all have a (String, Throwable) constructor at the moment?

The only problem to have some base exception for all types is the AssertionError in the hierarchy as already noted in issue #4.

Having a common base exception would be one way to solve it but not the only one. Your proposal in #21 (DebugInformation) would be another possibility, right?

[mmichaelis]

Exit States: Failure/Skipped/Error (Success)

(Forgot to mention the Success state of course.) Actually I relied on the interpretation of CIs like Jenkins and IDEs like IntelliJ Idea. And I think the four exit states are actually useful especially in case of integration tests where a test setup might be somewhat complex and it's important to distinguish if a feature is broken or the feature could not be tested because the test preparation failed. If this isn't the case in JUnit4 I suggest to discuss this again. But this should be an extra issue.

About: Common Constructor

Currently the AssertionFailedError has some special constructors for actual and expected value which the others don't share. But you are right, the suggested DebugInformation in #21 would be a solution. I am currently working on a draft... but it takes some time.

[marcphilipp]

Ok, let us know when you're ready! :)

[mmichaelis]

Not yet ready... but I have a proof-of-concept ready which could already benefit from some feedback:

https://github.com/mmichaelis/opentest4j/tree/issue-21-introduce-debug-information

From the commit message:

---

Issue #21: Introduce DebugInformation

Introducing a more generic way to provide additional information for debugging upon assertion failures. The DebugInformation is key-value based and can contain actual any information.

In addition to this extended the ValueWrapper (now renamed to ValueDescriptor as it describes the value) so that it does some effort to serialize objects applying some strategies. This way we
could also provide images as for example for UI tests.

The current state neither includes documentation, more tests and misses the commonly used exception for all exceptions provided inside here.

This approach also addresses issues:

• Support for Exception-Creation-Strategies: AssertionFailedException, TestAbortedException, ... #20 support for exception creation strategies
• Detach the exception hierarchy from AssertionError #4 detach AssertionError from exceptions here
• Define a standard for test reports #9 partially, as you the debug information provided here should eventually merge into a standard test report.

---

Some Notes:

• org.opentest4j.MultipleFailuresErrorTests: fails on Windows because the exception and the test build the messages in different ways. String.format() (as used in test) uses carriage-return-linefeed on windows, while the exception only uses linefeed in its message. Would have fixed it, if I know what the expectations is (so who is right: the test or the exception)?
• I started with a much richer DebugInformation for example also providing methods to add attachments (which again could make it into the common test report, see Define a standard for test reports #9). But after all refactorings this could even make it into a simple serializable value.

These changes also address #19 by not breaking with the Java behavior of cause-initialization -- but the solution is of course much different from that.

I will continue on that poc as soon as possible. But meanwhile I am happy to hear your feedback.