The benchmark reporting offers a mix of minimum repetition times, mean repetition times, with and without seeds depending on the benchmark.
It would be more useful to:
- record all runtimes during benchmarking to disk, for post processing
- Post-process benchamrk times to find min, mean, deviation etc.
- Seed runs for more deterministic benchmarking, though as models initialisation is stochastic and initialisation RNG varies multiple seeds should still be benchmarked
The benchmark reporting offers a mix of minimum repetition times, mean repetition times, with and without seeds depending on the benchmark.
It would be more useful to: