Andrus wrote:
> Well I was thinking more about adding a 3rd failure state:
>
> 1. Error - something that failed unexpectedly (no assertions in the tests)
> 2. Failure - something that failed in the assertion code
> 3. "Soft Failure" - a special assertion *for bug tracking only*.
4. Expected Failure - test test case will fail but has an assertion that
a certain error/exception will occur.
To reuse Holger's example:
void testSelectObjectForIdNullArg()
{
boolean didCatchNPE = false;
try
{
SelectQuery q = QueryHelper.selectQueryForObjectId(null);
Assert.fail(); // bail out here if no error occurred
}
catch (NullPointerException npe)
{
didCatchNPE = true;
}
Assert.assertTrue(didCatchNPE);
}
This way, if all the tests are green, we know that we still have all of
the nonresolved bugs in the codebase. If the testcase turns red, we know
that an error "went away" and we have to dig deeper into this issue.
It's a little bit reverting the logic of JUnit but then again we don't
have to jump around in loops. I'd put all the tests for known bugs in
their own testsuite so we can clearly separate the good testcases from
the bad ones.
> (2) and (3) are similar, but (2) will generate a JVM exit status of "1" so that the nightly build Perl script would fail, while (3) will generate an
exit status of "0".
We had a similar nuisance with ant exiting via System.exit(1) in one of
our Daedalos projects. The fix for that situation was to carry around a
patched version of the specific class in our project. But I wouldn't ask
for that kind of trouble ....
-dirk
This archive was generated by hypermail 2.0.0 : Mon Dec 23 2002 - 01:07:07 EST