RFC: Mocked Timer APIs

[jamiebuilds]

Writing this up to go along with #1825, because it's another major feature that I think Ava could include and improve upon by baking it into the framework.

Why add timer mocking?

Timer mocking is important for:

• Test reliability
• Test performance

Why not a separate library?

• Because building it in can help improve the performance of Ava tests
• Because existing tools are so complex people don't want to use them
• Because Ava can provide a better experience by building it into the test framework
  • By cleaning timers up automatically
  • By providing error messages when timers are left running and the test has finished

Examples

const test = require('ava');

test('timers', async t => {
  let c = t.clock();
  let called = 0;

  setTimeout(() => called++, 1000);
  await c.time(1000); // move timers ahead in milliseconds
  t.is(called, 1);

  process.nextTick(() => called++);
  await c.tick(1); // move micro-tasks ahead by "ticks"
  t.is(called, 2);

  requestAnimationFrame(() => called++); // when using JSDom
  await c.frame(1); // move frames ahead
  t.is(called, 3);

  requestAnimationFrame(() => {
    setTimeout(() => {
      process.nextTick(() => {
        Promise.resolve().then(() => {
          called++;
        });
      });
    }, 1000);
  });

  // run through all active timers/microtasks/frames until
  // theres nothing left in the queue.
  await c.drain();
  t.is(called, 4);

  // you don't have to restore the clock at the end of the test,
  // it will happen automatically
});

API type definition

interface Clock {
  time(ms: number = 1): Promise<void>;
  tick(ticks: number = 1): Promise<void>;
  frame(frames: number = 1): Promise<void>;
  drain(): Promise<void>;
}

interface t {
  clock(opts?: {}): Clock;
}

How to implement?

SinonJS has already done the work of splitting out their timer faking to a separate package:

https://github.com/sinonjs/lolex

It's already usable within Ava, but I think that we could improve the experience further.

Why await c.time/tick/frame()?

Right now it's impossible to completely mock out promises because of async-await. The fix is to wait a microtask without actually clearing it.

This does mean that we aren't completely mocking away timers, but it might actually be a good thing in case code does depend on microtasks not being run immediately.

[novemberborn]

Ava can provide a better experience by building it into the test framework

• By cleaning timers up automatically
• By providing error messages when timers are left running and the test has finished

Could you elaborate on cleanup and how it relates to concurrent tests?

We have some open issues to prevent exiting while the event loop is busy, and to enable us to fail a test down the line even if all assertions pass, e.g. due to a leaking timer.

[jamiebuilds]

Could you elaborate on cleanup and how it relates to concurrent tests?

I hadn't fully considered the concurrency problem because building this in requires you to override globals which could be used by any concurrent test.

I'm not sure it's even possible to (fully) detect what timer was created by what test. The closest thing I've got is looking at the stack trace, and that has all sorts of problems.

The other option here is always mocking timers. Which would be good for performance, but (based on Jest's early state) really bad for developer experience. Maybe this is the sort of thing opted into on a file basis:

const test = require('ava');
const c = test.clock(); // must be called immediately?

test('timers', async t => {
  let called = 0;

  setTimeout(() => called++, 1000);
  await c.time(1000); // move timers ahead in milliseconds
  t.is(called, 1);
});

[novemberborn]

I wonder if there's an extension of #1456 that might be useful here. Just sketching:

// clock.js
import test from 'ava'

const withClock = test.serial.use(async (t, next) => {
  const {destroy, ...fns} = createClock()
  try {
    await next({...t, ...fns})
  } finally {
    destroy()
  }
})

export default withClock

import test from './clock'

test('timers', async t => {
  let called = 0;

  setTimeout(() => called++, 1000);
  await t.time(1000); // move timers ahead in milliseconds
  t.is(called, 1);
});

That is to say, perhaps we can create new test interfaces that have specific behavior, and that can decorate the execution context. They could force synchronous execution, etc.

There's a bit of #222 in here, in that it'd make sense to allow beforeEach etc. Since those issues were last updated the runner code has been refactored making this more feasible.

[jamiebuilds]

That is to say, perhaps we can create new test interfaces that have specific behavior, and that can decorate the execution context. They could force synchronous execution, etc.

Hmm... I would almost prefer a way of saying that "this test needs to be run serially" as part of creating the clock. Otherwise you'd almost certainly end up with multiple test functions.

const test = require('ava');
const testWithClock = require('../../test-helpers/ava-with-clock');

vs

test('timers', async t => {
  let c = await t.clock();
  // Error: Cannot call t.clock() in tests that run concurrently (use `test.serial`)
});

---

Another idea: If we could hook into the test scheduler, you could make t.clock() return a promise that resolves once other concurrent tests are complete and make sure that other tests aren't running at the same time.

const test = require('ava');

test('timers', async t => {
  let c = await t.clock();
  let called = 0;

  setTimeout(() => called++, 1000);
  await c.time(1000); // move timers ahead in milliseconds
  t.is(called, 1);
});

Since some test files can end up being hundreds of tests (where concurrency has a huge impact), it would be good if you didn't need to make every test in the file run serially.

[novemberborn]

If we could hook into the test scheduler, you could make t.clock() return a promise that resolves once other concurrent tests are complete and make sure that other tests aren't running at the same time.

Oh I like that! Perhaps await t.waitSerial()? If used in a serial tests it resolves instantly, but in a concurrent test it waits for all non-waiting tests to complete before running the remaining ones serially again.

Then a decorator could implement t.clock() so the test becomes a serial one at runtime.

[SephReed]

If my code uses requestAnimationFrame, is AVA not the right tool to use?

[SephReed]

Nvm, got it:

const browserEnv = require('browser-env');
browserEnv([
  'window', 
  'setTimeout'
]);

global.requestAnimationFrame = (cb) => setTimeout(cb, 24);

[novemberborn]

With #2435 I'm exploring ways to create custom test() methods as well as adding custom assertions. I think that would provide a good foundation for the issues discussed here.