Skip to content

Default hooks to run concurrently, with explicit serial hooks possible #1684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
novemberborn opened this issue Feb 4, 2018 · 0 comments
Closed
Assignees

Comments

@novemberborn
Copy link
Member

Currently, hook execution subtly differs from test execution. Hooks always execute sequentially, even if they're not marked as being serial.

Prompted by #1178, hooks are removed from test.serial in #1670. I think we should keep such hooks, but they should work properly:

  • Hooks always execute in the order they're defined (by category, e.g. before, afterEach.always, etc)
  • beforeEach, afterEach and afterEach.always hooks execute for each test, when the test executes
    • Serial tests are executed before concurrent ones, if any
    • Concurrent tests are executed after serial ones, if any, without regard for the state of other concurrent tests
    • For each test, beforeEach hooks are executed after before hooks, if any
    • For each test, afterEach hooks are executed after the test has finished
    • For each test, afterEach.always hooks are executed after afterEach hooks, if any
    • For serial tests, the beforeEach hooks are executed after the previous serial test's afterEach.always hook (or afterEach hook, or test itself, if any)
  • after hooks are executed after all tests have finished
  • after.always hooks are executed after after hooks
  • Unlike tests, serial hooks are not reordered to run first
  • Serial hooks are executed after preceding hooks in their category have finished
  • Concurrent hooks execute concurrently, without regard for the state of other concurrent hooks (in their category)

Here, "serial" means tests and hooks explicitly marked as serial. "Concurrent" means tests and hooks not marked as serial, that execute asynchronously either by using an asynchronous method, returning a promise or observable, or using the cb interface. Synchronous tests not explicitly marked as serial may execute concurrently with previous concurrent test, even though they finish immediately.

This proposal lets us bring back hooks to test.serial, but now their execution will be properly understood. It might break some tests with multiple concurrent hooks in each category, but I expect that to be rare.

@novemberborn novemberborn self-assigned this Feb 4, 2018
novemberborn added a commit that referenced this issue Feb 10, 2018
Fixes #1684. Fixes #1416. Refs #1158.

Properly support serial hooks. Hooks are divided into the following
categories:

* before
* beforeEach
* afterEach
* afterEach.always
* after
* after.always

For each category all hooks are run in the order they're declared in.
This is different from tests, where serial tests are run before
concurrent ones.

By default hooks run concurrently. However a serial hook is not run
before all preceding concurrent hooks have completed. Serial hooks are
never run concurrently.

Always hooks are now always run, even if --fail-fast is enabled.

Internally, TestCollection, Sequence and Concurrent have been removed.
This has led to a major refactor of Runner, and some smaller breaking
changes and bug fixes:

* Unnecessary validations have been removed
* Macros can be used with hooks
* A macro is recognized even if no additional arguments are given, so it
can modify the test (or hook) title
* --match now also matches todo tests
* Skipped and todo tests are shown first in the output
* --fail-fast prevents subsequent tests from running as long as the
failure occurs in a serial test
novemberborn added a commit that referenced this issue Feb 10, 2018
Fixes #1684. Fixes #1416. Refs #1158.

Properly support serial hooks. Hooks are divided into the following
categories:

* before
* beforeEach
* afterEach
* afterEach.always
* after
* after.always

For each category all hooks are run in the order they're declared in.
This is different from tests, where serial tests are run before
concurrent ones.

By default hooks run concurrently. However a serial hook is not run
before all preceding concurrent hooks have completed. Serial hooks are
never run concurrently.

Always hooks are now always run, even if --fail-fast is enabled.

Internally, TestCollection, Sequence and Concurrent have been removed.
This has led to a major refactor of Runner, and some smaller breaking
changes and bug fixes:

* Unnecessary validations have been removed
* Macros can be used with hooks
* A macro is recognized even if no additional arguments are given, so it
can modify the test (or hook) title
* --match now also matches todo tests
* Skipped and todo tests are shown first in the output
* --fail-fast prevents subsequent tests from running as long as the
failure occurs in a serial test
novemberborn added a commit that referenced this issue Feb 10, 2018
Fixes #1684. Fixes #1416. Refs #1158.

Properly support serial hooks. Hooks are divided into the following
categories:

* before
* beforeEach
* afterEach
* afterEach.always
* after
* after.always

For each category all hooks are run in the order they're declared in.
This is different from tests, where serial tests are run before
concurrent ones.

By default hooks run concurrently. However a serial hook is not run
before all preceding concurrent hooks have completed. Serial hooks are
never run concurrently.

Always hooks are now always run, even if --fail-fast is enabled.

Internally, TestCollection, Sequence and Concurrent have been removed.
This has led to a major refactor of Runner, and some smaller breaking
changes and bug fixes:

* Unnecessary validations have been removed
* Macros can be used with hooks
* A macro is recognized even if no additional arguments are given, so it
can modify the test (or hook) title
* --match now also matches todo tests
* Skipped and todo tests are shown first in the output
* --fail-fast prevents subsequent tests from running as long as the
failure occurs in a serial test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant