Problem Statement
Summary
When the nightly benchmark workflow is triggered against an llm-d PR (via /test / workflow_dispatch), the Checkout Code step correctly checks out the PR ref — but a later step (Conditionally clone llm-d from main repo) immediately does git reset --hard origin/main on that same checkout, discarding the PR's changes. As a result the run executes against origin/main's guide content, not the PR's, so any change under guides/** (manifests, kustomize overlays, guide.yaml, README) is silently reverted and never exercised.
Root cause
.github/workflows/reusable-ci-nightly-benchmark.yaml
if [[ -d ~/_work/llm-d/llm-d ]]; then
ln -s ~/_work ~/work # symlink → same tree the PR was checked out into
fi
if [[ ! -d ~/work/llm-d ]]; then
git clone https://github.com/llm-d/llm-d.git
else
# Refresh the persistent checkout so it doesn't run against stale guide config.
git -C ~/work/llm-d/llm-d fetch --prune origin
git -C ~/work/llm-d/llm-d reset --hard origin/main # ← discards the PR checkout
...
fi
On the persistent self-hosted runner, ~/work/llm-d/llm-d is the PR checkout (via the ~/_work → ~/work symlink), so reset --hard origin/main overwrites the PR's tree with main.
Impact
The reset --hard origin/main discards the entire checked-out PR tree, but the nightly only consumes guides/** from that checkout at runtime (kustomize repoPath, guide manifests/overlays/RBAC/router values/env.sh). So any llm-d PR changing guide content cannot be validated by this nightly pre-merge — it always runs against origin/main's guides
Evidence
https://github.com/llm-d/llm-d/actions/runs/30385519489
Proposed Solution
Proposed fix
Make the refresh target the ref under test instead of hardcoding origin/main. Options:
- When triggered for a PR (github.event_name / a passed head SHA/ref), reset --hard to that ref rather than origin/main; only fall back to origin/main for scheduled runs.
Alternatives Considered
No response
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response
Problem Statement
Summary
When the nightly benchmark workflow is triggered against an llm-d PR (via /test / workflow_dispatch), the Checkout Code step correctly checks out the PR ref — but a later step (
Conditionally clone llm-d from main repo) immediately doesgit reset --hard origin/mainon that same checkout, discarding the PR's changes. As a result the run executes against origin/main's guide content, not the PR's, so any change under guides/** (manifests, kustomize overlays, guide.yaml, README) is silently reverted and never exercised.Root cause
.github/workflows/reusable-ci-nightly-benchmark.yamlOn the persistent self-hosted runner, ~/work/llm-d/llm-d is the PR checkout (via the ~/_work → ~/work symlink), so reset --hard origin/main overwrites the PR's tree with main.
Impact
The
reset --hard origin/maindiscards the entire checked-out PR tree, but the nightly only consumes guides/** from that checkout at runtime (kustomize repoPath, guide manifests/overlays/RBAC/router values/env.sh). So any llm-d PR changing guide content cannot be validated by this nightly pre-merge — it always runs against origin/main's guidesEvidence
https://github.com/llm-d/llm-d/actions/runs/30385519489
Proposed Solution
Proposed fix
Make the refresh target the ref under test instead of hardcoding origin/main. Options:
Alternatives Considered
No response
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response