Skip to content

Commit 461db39

Browse files
FrancescAltedclaude
andcommitted
Run five interleaved bench rounds instead of three
At three rounds one cell (macos-latest / 3.14) flagged where() at 1.287x along with the other large-array compute benchmarks, while the call-overhead-bound benchmarks on that same cell -- compress2 of 8 KB buffers at 0.986x, SChunk attribute access at 1.040x -- were clean. That is backwards: abi3 taxes the Python/C boundary, so a cost has to appear on the call-heavy benchmarks before it appears on 4M-element vectorized compute. The same where() benchmark came in at 0.987x, 1.009x and 0.971x on the other three cells, and 0.969x locally. So this is a noisy shared runner, not an ABI effect. Five rounds gives the per-benchmark minimum a larger sample to draw an uncontaminated observation from, which is the point of using min as the estimator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 14b621e commit 461db39

1 file changed

Lines changed: 7 additions & 2 deletions

File tree

.github/workflows/abi3-bench.yml

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,13 +99,18 @@ jobs:
9999
done
100100
101101
# Interleaved so a slow patch on a shared runner hits both builds rather
102-
# than biasing whichever ran first.
102+
# than biasing whichever ran first. Five rounds rather than three: at
103+
# three, one macOS/3.14 cell flagged the 4M-element compute benchmarks
104+
# (where() at 1.287x) while the call-overhead-bound ones on the same cell
105+
# were clean -- backwards from how abi3 can possibly work, and contradicted
106+
# by the same benchmark on four other cells. More rounds gives `min` a
107+
# larger sample to find an uncontaminated one in.
103108
- name: Run interleaved benchmark rounds
104109
shell: bash
105110
run: |
106111
mkdir -p results/bench-${{ matrix.os }}-${{ matrix.python-version }}
107112
OUT=results/bench-${{ matrix.os }}-${{ matrix.python-version }}
108-
for round in 1 2 3; do
113+
for round in 1 2 3 4 5; do
109114
for v in abi3 base; do
110115
if [ -x "venv-$v/bin/python" ]; then
111116
PY="venv-$v/bin/python"

0 commit comments

Comments
 (0)