[RFC] each and matrix tests

[mrginglymus]

Summary

The new CSF testing feature (#30119) adds a familiar testing syntax to CSF factories, allowing users to write tests in storybook.

It's a huge improvement and a welcome feature, but inevitably there are features that other testing frameworks have that are not yet present.

This RFC covers the lack of .each (or similar) functionality, to allow for easily parameterised tests.

Problem Statement

A storybook component is likely to have several parameters to configure its appearance and behaviour.

Before CSF tests, for chromatic users, this meant a lot of copy + pasting to achieve the required combinatorial set of stories.

CSF tests allow for a slightly more refined approach, but owing to the static analysis requirement for indexing, the usual approach of dynamically generating variations at runtime is not possible.

Non-goals

No response

Implementation

This RFC proposes the addition of two new test factories on the meta object.

The each factory will accept a list of parameters to be applied to tests:

/**
 * This will run the test function with
 * p1: 'a', p2: 1
 * p1: 'b', p2: 2
 */
Default.each(
  'Test name',
  [
    ['a', 1],
    ['b', 2]
  ],
  (context, p1, p2) => {}
)

The matrix factory will accept a list of parameters to generate combinations to be applied to tests

/**
 * This will run the test function with
 * p1: 'a', p2: 1
 * p1: 'a', p2: 2
 * p1: 'b', p2: 1
 * p1: 'b', p2: 2
 */
Default.matrix(
  'Test name',
  [
    ['a', 'b'],
    [1, 2]
  ],
  (context, p1, p2) => {}
)

The parameters would also be available to customise the story options, so that they can be used to customise story args and parameters:

Default.each(
  'Test name',
  [
    ['a', 1],
    ['b', 2]
  ],
  (p1, p2) => ({
    args: {
      children: p1
  }),
  (context) => {} //test
)

Note that the APIs here are somewhat speculative - the factories could easily be any of the following:

// Entry point
Default.test.each();
Default.testEach();

// Calling
Default.each('Name', [])(() => {}); // similar to vitest

and many more

Indexing

In order to ensure tests are distinguishable, users will be able to need to inject parameters into them. However, as the test names must be statically analysable, this cannot be done at run time. A simple solution is to use a sprintf like syntax:

/**
 * Will generate tests:
 *   - 'Testing with a'
 *   - 'Testing with b'
 */
Default.each(
  'Testing with %s',
  [['a'], ['b']],
  () => {}
)

Note that this also requires the parameters to be inline rather than references.

Prior Art

The each method is an fairly common pattern in testing libraries:

• jest
• vitest

even though the same could often be achieved with a simple for loop

The matrix method is less common - again, due to the dynamic nature of most test runners, the combinations can simple be passed to each. However, it's quite common in CI:

• Github Actions
• Jenkins

In addition, chromatic offers story modes which offer a similar functionality, but only at the global level.

Deliverables

No response

Risks

No response

Unresolved Questions

• Entrypoints to api (top level, child of .test, etc)
• Shape of calling function (single function, function factory)

Alternatives considered / Abandoned Ideas

For runtime, it is technically possible to return a custom preview that will generate custom meta to allow for custom runtime objects, though this would be extremely fragile.

For indexing, experimental_indexers would allow detection of these stories; however, this may lead to a fair amount of duplication.