Test harness crashes/hangs on test crash

[rmacnak-google]

import "package:test/test.dart";
import "dart:ffi";

main() {
  test("fault", () {
    Pointer.fromAddress(0).cast<Long>()[0] = 0;
  });
}

With dart test example.dart, the test harness itself crashes.

With dart test -c exe example.dart, the test harness hangs.

[jakemac53]

With dart test example.dart, the test harness itself crashes.

In this mode we run the test in an isolate spawned by the main test runner (it is given a kernel file). Is the exception thrown in this case something we could catch? I am not sure how exactly that would work.

With dart test -c exe example.dart, the test harness hangs.

This case we should be able to handle better I think.

[rmacnak-google]

Is the exception thrown in this case something we could catch?

There is no exception. The process terminates with SIGSEGV or SIGABRT. The test harness and the tests need to be in separate processes to be robust.

[jakemac53]

There is no exception. The process terminates with SIGSEGV or SIGABRT. The test harness and the tests need to be in separate processes to be robust.

Yeah ok then this is basically just working as expected today.

We could run all tests in a separate process instead of in an isolate, I am not sure if there would be a performance impact to that. This would however have an extra benefit of exit() calls not exiting the entire test runner (although it would still exit the suite). But, we could detect early exits in the runner and make them failures.

[jakemac53]

cc @natebosch

[natebosch]

I think moving vm tests out of process by default is unlikely to get prioritized. Crashes are not a common scenario and the negative side effects are limited to a slightly confusing UX and not seeing the results of some tests that might have run if the test runner was still live. It doesn't turn a failure into a incorrect success, nor cause spurious failures. The exit loophole is a bit riskier since it can make failures look like successes with a 0 exit code.

The hang we should address.

[jakemac53]

Fwiw, we can override exit using IOOverrides as well.