Failure in a benchmark iteration is not reported

[Akirathan]

For example, here: https://github.com/enso-org/enso/actions/runs/8117569264/job/22190038426?pr=9150#step:8:16302.
There is a failure in a benchmark iteration, but the whole benchmark run succeeds. We should report an error once something like that occurs.

Dry-run benchmarks run as part of our tests should fail when some of the benchmark iteration fails, whereas, the real benchmarks should not.

Relates to: #9394

[Akirathan]

Priority is p-high, as it causes us not to observe recent benchmark failures. It requires a manual output reading.

[Akirathan]

If Enso dry-run benchmark fails, the process still succeeds. For example, at https://github.com/enso-org/enso/actions/runs/8340141213/job/22823504898?pr=9458#step:8:2260. Only JMH dry-run benchmarks were fixed. Reopening.

GitHub

Diff editor · enso-org/enso@858d20f

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

[Akirathan]

If Enso dry-run benchmark fails, the process still succeeds. For example, at https://github.com/enso-org/enso/actions/runs/8340141213/job/22823504898?pr=9458#step:8:2260. Only JMH dry-run benchmarks were fixed. Reopening.

GitHub**Diff editor · enso-org/enso@858d20f**Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

The benchmark that silently fails in that case is

enso/test/Benchmarks/src/Vector/Map_Error_Benchmark.enso

Line 62 in f2cb1f0

map_id_always_throw data.value on_problems=problem_behavior . catch

which catches all the errors thrown within the benchmarking function. So the silent failure is a feature for that particular benchmark, not a bug. Closing this issue once again.

GitHub

Diff editor · enso-org/enso@858d20f

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.