Skip to content

[Feature]: reusable-ci-nightly-benchmark.yaml resets llm-d to origin/main, so changes in an llm-d PR can't be /test-validated before merge #233

Description

@aavarghese

Problem Statement

Summary

When the nightly benchmark workflow is triggered against an llm-d PR (via /test / workflow_dispatch), the Checkout Code step correctly checks out the PR ref — but a later step (Conditionally clone llm-d from main repo) immediately does git reset --hard origin/main on that same checkout, discarding the PR's changes. As a result the run executes against origin/main's guide content, not the PR's, so any change under guides/** (manifests, kustomize overlays, guide.yaml, README) is silently reverted and never exercised.

Root cause

.github/workflows/reusable-ci-nightly-benchmark.yaml

if [[ -d ~/_work/llm-d/llm-d ]]; then
  ln -s ~/_work ~/work        # symlink → same tree the PR was checked out into
fi
if [[ ! -d ~/work/llm-d ]]; then
  git clone https://github.com/llm-d/llm-d.git
else
  # Refresh the persistent checkout so it doesn't run against stale guide config.
  git -C ~/work/llm-d/llm-d fetch --prune origin
  git -C ~/work/llm-d/llm-d reset --hard origin/main   # ← discards the PR checkout
  ...
fi

On the persistent self-hosted runner, ~/work/llm-d/llm-d is the PR checkout (via the ~/_work → ~/work symlink), so reset --hard origin/main overwrites the PR's tree with main.

Impact

The reset --hard origin/main discards the entire checked-out PR tree, but the nightly only consumes guides/** from that checkout at runtime (kustomize repoPath, guide manifests/overlays/RBAC/router values/env.sh). So any llm-d PR changing guide content cannot be validated by this nightly pre-merge — it always runs against origin/main's guides

Evidence

https://github.com/llm-d/llm-d/actions/runs/30385519489

Proposed Solution

Proposed fix

Make the refresh target the ref under test instead of hardcoding origin/main. Options:

  1. When triggered for a PR (github.event_name / a passed head SHA/ref), reset --hard to that ref rather than origin/main; only fall back to origin/main for scheduled runs.

Alternatives Considered

No response

Willingness to Contribute

Yes, I can submit a PR

Additional Context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions