Skip to content
Performance

parity: batch 49 (contextual declared flow reports) — snapshots and r… #3584

Sign in to view logs

parity: batch 49 (contextual declared flow reports) — snapshots and r…

parity: batch 49 (contextual declared flow reports) — snapshots and r… #3584

Workflow file for this run

.github/workflows/perf.yml at 27896f5
# The performance gate from docs/adr/0009-performance-gate.md, plus the
# conformance rates.
#
# The two travel together because both answer "where is the port?" rather than
# "did this change break something", and both are worth a reading on a quiet week.
# The artifact carries every suite's snapshot, so a download shows the full
# per-case detail and not just the summary line.
#
# Separate from ci.yml because it needs a Go toolchain and a typescript-go build,
# and because it is the one job whose result is a *measurement* rather than a
# pass/fail on our own source. Its artifacts are the point: the thresholds catch
# cliffs, the recorded ratios catch drift.
name: Performance
on:
push:
branches: [main]
pull_request:
# Comparing against upstream means the number can move without our code moving,
# so it is worth taking a reading even on a quiet week.
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:
env:
CARGO_TERM_COLOR: always
# A benchmark racing another benchmark measures the contention, not the code —
# and racing anything else measures it just as badly. The group is shared with
# `ci.yml`, which holds the same runner label, and is deliberately not keyed on
# the ref: two refs benchmarking at once contend exactly as much as two jobs on
# one ref do.
#
# `cancel-in-progress` was `true` here until 2026-08-05 and was part of the
# problem rather than the cure. Cancelling a superseded perf run does not stop
# its `Conformance rates` step — release, rayon, 12,444 cases, `if: always()` —
# from saturating the host while the replacement run starts benchmarking, and
# our benchmark is the *first* thing `xtask perf` does. Queueing is what was
# actually wanted. See `bd tsr-1xm` for the measurement.
concurrency:
group: self-hosted-exclusive
cancel-in-progress: false
jobs:
compare:
name: vs typescript-go
runs-on: [self-hosted, Linux, X64]
steps:
- uses: actions/checkout@v4
with:
# Recursive for the benchmark fixtures, which resolve through the
# TypeScript submodule inside typescript-go; shallow because no history
# is read. See the same note in ci.yml.
submodules: recursive
fetch-depth: 1
# Same pin as ci.yml and rust-toolchain.toml — a benchmark compiled by a
# different rustc than the gates is not comparing the same compiler.
- uses: dtolnay/rust-toolchain@1.96.0
- uses: Swatinem/rust-cache@v2
- uses: actions/setup-go@v5
with:
# Taken from the pinned submodule rather than hard-coded, so a pin bump
# that raises the Go requirement does not silently build with the wrong
# toolchain.
go-version-file: vendor/typescript-go/go.mod
cache-dependency-path: vendor/typescript-go/go.sum
# Build both sides before measuring. A compile running alongside a benchmark
# is the single largest source of noise on a shared runner — locally it was
# enough to move one fixture from 0.71x to 1.04x.
- name: Build tsr
run: |
cargo build --release -p tsr-parser --example rss
cargo build --release -p tsr-parser --bench parse
- name: Build typescript-go
working-directory: vendor/typescript-go
run: go test -run '^$' -bench '^$' ./internal/parser/
# A reporting-only build failure must not suppress the existing gate.
# Check outcome (before continue-on-error) before using cached binaries.
- name: Build whole-project CLIs
id: build_whole_project
continue-on-error: true
run: |
cargo build --release --jobs 1 --bin tsr
go -C vendor/typescript-go build -p 1 -buildvcs=false -o ../../target/perf-tsgo ./cmd/tsgo
- name: Compare
id: compare
run: cargo run -p xtask -- perf
- name: Check whole-project reporting controls
if: always()
run: |
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover \
-s scripts -p 'test_*whole_project_perf.py'
# Reporting is independently useful while work/input telemetry and a
# representative suite are incomplete. The tiny public project exercises
# plumbing, not the <=0.50 release target or a timing regression ratchet.
# Both modes run serially under the existing exclusive runner group.
- name: Whole-project observations
if: always()
continue-on-error: true
run: |
PYTHONDONTWRITEBYTECODE=1 python3 scripts/report_whole_project_perf.py \
--project benches/projects/generic-imports/tsconfig.json \
--tsr target/release/tsr --tsgo target/perf-tsgo \
--output-dir whole-project-results --samples 5 --timeout 60 \
--builds-ready '${{ steps.build_whole_project.outcome == 'success' }}'
# The conformance rates ride along with the perf artifacts so one download
# answers "where is the port?" without reading a log. `ci.yml` also runs the
# corpus, but as a *ratchet* — it fails when the committed snapshots move.
# This publishes the numbers themselves, including on a scheduled run when
# nothing changed, which is the point: an artifact you can open and read.
#
# Release, and after the perf comparison. Debug is roughly 10x slower here
# and needs a 32 MiB worker stack (see the constant in the coverage binary);
# running it before the benchmark would also be the compile-next-to-a-
# benchmark noise this job takes care to avoid.
- name: Conformance rates
if: always()
run: |
cargo run --release -p tsr-conformance --bin coverage \
| tee conformance-summary.txt
# `always()`: a failing gate is exactly when the numbers are worth keeping.
- name: Publish results
if: always()
uses: actions/upload-artifact@v4
with:
name: perf-results-${{ github.sha }}
path: |
perf-results.json
perf-summary.md
conformance-summary.txt
whole-project-results/
crates/tsr-conformance/snapshots/
if-no-files-found: warn
retention-days: 90
- name: Job summary
if: always()
run: |
cat perf-summary.md >> "$GITHUB_STEP_SUMMARY" || true
cat whole-project-results/summary.md >> "$GITHUB_STEP_SUMMARY" || true
{
echo
echo '## Conformance'
echo
echo '```'
cat conformance-summary.txt
echo '```'
} >> "$GITHUB_STEP_SUMMARY" || true