Real-world benchmarks (sturdy)

[sjakobi]

In #282 (comment) @svenkeidel mentioned the sturdy project:

My collaborator Tobias Hombücher is currently working on a static analysis for Scheme that uses this hashmaps and hashsets extensively (https://gitlab.rlp.net/plmz/sturdy/-/tree/PowersetT/scheme). We have a benchmark suit with a few scheme programs (https://gitlab.rlp.net/plmz/sturdy/-/blob/PowersetT/scheme/bench/FixpointBench.hs). Once Tobias gets the analysis working, this would be a great real-world benchmark for the unordered-containers library, because it uses the insert, delete, union and isSubmapOf functions.

I wonder whether it would be worth adopting parts of sturdy as a benchmark suite, or maybe just document how to use it as a benchmark for our code.

[thombuecher]

Dear @sjakobi,

I am a collaborator of @svenkeidel and just updated the benchmarks of our static analysis for Scheme, so that all benchmarks that currently perform as intended can be executed. You can checkout the PowersetT branch (https://github.com/svenkeidel/sturdy/tree/PowersetT) at commit 34535e40c925905ea5b74210de64469967973f09 (svenkeidel/sturdy@34535e4) and run the benchmarks with stack bench sturdy-scheme.

Regards,
Tobias

[sjakobi]

Thanks for the instructions, @thombuecher! :)

Benchmarking a custom unordered-containers should then be as simple as adding another extra-dep to the stack.yaml.

Benchmark results could be compared for example with criteriton-compare.

BTW, I noticed that the build took quite a bit of time. Re-building with a modified u-c should also be quite slow, in part due to the criterion dependency. Using gauge instead would probably speed that up quite a bit.

Now I think that it would be somewhat better to document this benchmark in a more permanent document than this random issue. I'm not quite sure where though.

Otherwise, I'll just hope we'll find this issue again when the need for a real-world benchmark comes up.

---

EDIT: I've also created a fork of sturdy at https://github.com/haskell-unordered-containers/sturdy that I hope will remain useful in case something happens to https://github.com/svenkeidel/sturdy.