Implement a minimal coverage strategy and make it the default

[swankjesse]

Today if you use a lot of Burst, you end up running a lot of tests!

Cartesian Product

A test with N,M parameters runs NxM tests. For example, a test with 3 sizes & 4 sodas runs 12 tests:

	Small	Medium	Large
Pepsi	✅	✅	✅
Coke	✅	✅	✅
Dr Pepper	✅	✅	✅
Root Beer	✅	✅	✅

Adding a choice of 3 distributions triples the number of tests to 36. With Cartesian product strategy, the number of tests multiply.

Minimal Coverage

I’d like to build a new coverage strategy that runs each option at least once.

A test with N,M parameters would run N+M-1 tests. The test above would run 6 tests:

	Small	Medium	Large
Pepsi	✅	✅	✅
Coke		✅
Dr Pepper		✅
Root Beer		✅

Adding a choice of 3 distributions adds only 2 more tests. (All existing tests get distribution=Can, plus Pepsi_Medium_Bottle, Pepsi_Medium_Fountain).

API

I think we express it like this:

@Burst(strategy = CartesianProduct)
class DrinkSodaTest { ... }

@Burst(strategy = MinimalCoverage)
class DrinkSodaTest { ... }

Once implemented I’d also like to explore making this option the default. I’d like to encourage people to Use More Burst and the Cartesian product strategy makes it expensive to do so.

[JakeWharton]

Thinking of all the places I've used parameterization over the years, I have a hard time believing the common case is when the axes of variation do not interact with each other. There definitely are cases where I've only needed minimal coverage as you've defined, but in most cases that I can think of, for me at least, I'm actively seeking the full cartesian product.

I would be pretty upset if manufacturing changed the paper composition of our large cups to save money, but then we only discover (in the field) that Dr Pepper dissolves it in seconds giving customers wet laps and our tests didn't cover large cups with Dr Pepper.

[swankjesse]

Yeah, maybe changing the default is bad.

In our Cash App tests we vary on accessibility text size & dark mode. These are independent!

[JakeWharton]

When I was thinking about my reply above, I actually came up with that as an example of a pair which I thought were independent. But then I convinced myself that sometimes they're not. A larger text size might change the layout such that different subcomponents are selected which may or may not react to theme correctly. I will admit that this is an extremely unlikely scenario, and in this case you could switch the strategy back.

An alternate proposal that I have thought about for no more than a few seconds would be to declare a particular parameter as being @Independent or something.