Simplify baseline names in snapshot testing

[bbannier]

We previously would include a running number for the specific test
parameterization in the snapshot for each sample test. This meant that
tests could get renamed whenever a new sample was added.

With this patch we now override the default name enumerating each
parameterization and instead set it to the name of the sample a test
runs on. With that snapshots should be much more stable.

[awelzel]

LGTM, only thought is if _get_samples() could return something like:

{
    "argvalues": [ ... ],
    "ids": [....],
}

And then **-splat it in the parameterize() call. Haven't tried, but that would make the naming/ordering local to _get_samples().

[bbannier]

{
"argvalues": [ ... ],
"ids": [....],
}

And then `**-splat` it in the parameterize() call. Haven't tried, but that would make the naming/ordering local to `_get_samples()`.

Purely from a typing standpoint, we'd need to return a similar but empty dict on the exception path for things to align, e.g.,

class Samples(TypedDict):
    argvalues: list[Path]
    ids: list[str]


def _get_samples() -> Samples:
   # ...
   try:
      # ...
        return {
            "argvalues": samples,
            "ids": [str(os.path.basename(s)) for s in samples],
        }
    except FileNotFoundError:
        return {"argvalues": [], "ids": []}

Would that look reasonable to you?

[awelzel]

Would that look reasonable to you?

Could we ditch FileNotFound error? Most likely empty samples is not what one would want to test anyhow?

I was thinking more about the following, but I'm now thinking your initial proposal with using ids= explicitly is better and easier to understand. Sorry for the noise.

def _get_samples():
    """Helper to enumerate samples"""

    # We exclude directories since we store snapshots along with the samples.
    # This assumes that there are no tests in subdirectories of `SAMPLES_DIR`.
    samples, ids = [], []
    for s in sorted([sample for sample in SAMPLES_DIR.iterdir() if sample.is_file]):
        samples += [s]
        ids += [s.parts[-1]]

    return {
        "argvalues": samples,
        "ids": ids,
    }

[bbannier]

Removed the handling of FileNotFound which can only be raised if SAMPLES_DIR does not exist, inserted an assertion instead.

I agree that the coupling between the internals of _get_samples (in particular: returns a sorted list), and its use is not nice, but like you wrote pulling all that into _get_samples itself doesn't make things much simpler either.