feat: add support for CRITERION_HOME (#7979)

This introduces a new CLI argument `--criterion-home` that we intercept
and use to set `CRITERION_HOME` for the bench runner. See comment in
code for details.

This allows us to specify the criterion root (home) where we start
looking for bench dirs, in order to avoid false positives from
`bazel-out/`.

---------

Co-authored-by: Bas van Dijk <[email protected]>

Author: nmattia

.github/workflows/schedule-rust-bench.yml

@@ -60,8 +60,11 @@ jobs:
           run: |
             echo -e "\e[34mRunning rust benchmarks on dedicated machine fr1-spm15!\e[0m"
 
+            CRITERION_HOME=$(mktemp -d)
+            trap "rm -rf $CRITERION_HOME" INT TERM EXIT
+
             while IFS= read -r tgt; do
-                bazel run "$tgt"
+                bazel run "$tgt" -- --criterion-home "$CRITERION_HOME"
             done < <(bazel query "attr(tags, 'rust_bench', ${{ matrix.target }})")
 
             while IFS= read -r bench_dir; do
@@ -82,6 +85,6 @@ jobs:
                     >report.json
                 curl --fail --retry 2 -sS -o /dev/null -X POST -H 'Content-Type: application/json' --data @report.json \
                     "https://elasticsearch.testnet.dfinity.network/ci-performance-test/_doc"
-            done < <(find -L ./bazel-out -type d -path '*/new')
+            done < <(find -L "$CRITERION_HOME" -type d -path '*/new')
 
             echo -e "\e[34mRust benchmarks on dedicated machine fr1-spm15 finished.\e[0m"

bazel/generic_rust_bench.sh

@@ -2,6 +2,30 @@
 
 set -euo pipefail
 
+# criterion supports specifying a "home" where it places its benchmark result tree.
+# at the time of writing, our bazel version (.bazelversion) is outdated and sh_binary
+# does not support `env_inherit` so add (or rather intercept) a new CLI argument
+# `--criterion-home` which we convert into `CRITERION_HOME`, which is read by
+# criterion.rs.
+#
+# https://github.com/criterion-rs/criterion.rs/blob/950c3b727a09d10067ea686e2ac6f1f23569168f/src/lib.rs#L142-L142
+
+# store passthru args
+args=()
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --criterion-home)
+            shift
+            export CRITERION_HOME="$1"
+            ;;
+        *)
+            # passthru
+            args+=("$1")
+            shift
+            ;;
+    esac
+done
+
 # When Cargo runs benchmarks, it passes the --bench or --test command-line arguments to
 # the benchmark executables. Criterion.rs looks for these arguments and tries to either
 # run benchmarks or run in test mode. In particular, when you run cargo test --benches
@@ -11,7 +35,7 @@ set -euo pipefail
 # present, or when --bench and --test are both present.
 #
 # https://bheisler.github.io/criterion.rs/book/faq.html#when-i-run-benchmark-executables-directly-without-using-cargo-they-just-print-success-why
-CMD="${BAZEL_DEFS_BENCH_PREFIX}${BAZEL_DEFS_BENCH_BIN} --bench $@"
+CMD="${BAZEL_DEFS_BENCH_PREFIX}${BAZEL_DEFS_BENCH_BIN} --bench ${args[@]}"
 
 echo "running ${CMD}"
 ${CMD}