Skip to content

ci: resolve latest Opus model dynamically for Claude review#278

Closed
fan-zhang-sv wants to merge 1 commit into
masterfrom
ci/claude-review-dynamic-model
Closed

ci: resolve latest Opus model dynamically for Claude review#278
fan-zhang-sv wants to merge 1 commit into
masterfrom
ci/claude-review-dynamic-model

Conversation

@fan-zhang-sv
Copy link
Copy Markdown
Collaborator

Summary

  • Replace the hardcoded --model claude-opus-4-6-default in the Claude Code review workflow with a dynamically resolved model
  • New pre-step queries the LLM Gateway's /v1/models endpoint, filters for clean Opus aliases matching claude-opus-N(-M)? (excludes -default, [1m], #priority, and non-Opus families), sorts with sort -V, and picks the highest
  • Falls back to claude-opus-4-6 if the gateway is unreachable, returns a non-2xx, returns malformed JSON, or yields no matches — so the workflow stays functional under all failure modes

Why

The reviewer was pinned to a single Opus string, so picking up newer families (e.g., claude-opus-4-7, claude-opus-5) required a workflow edit. This makes the version tracking automatic while still letting us manually bump the safety-net fallback.

Relevant context:

  • LLM Gateway model aliases doc — within-family updates are automatic; cross-family is manual
  • GENAI-1809 — Anthropic retirement cadence means pinning can silently go stale
  • The gateway hostname is already in the harden-runner egress allowlist, so no network policy changes needed

Behavior

Scenario Result
Gateway returns list with claude-opus-4-6, ...-default, Sonnet, GPT Picks claude-opus-4-6 (filters out -default and non-Opus)
Gateway returns future claude-opus-5-1 Picks claude-opus-5-1 automatically
Gateway times out / 5xx / auth fails Falls back to claude-opus-4-6, emits ::warning::
Gateway returns empty data: [] Falls back to claude-opus-4-6, emits ::warning::

Verified the jq filter + sort -V pipeline locally against mock gateway responses (including multi-family, suffixed aliases, and empty cases).

Test plan

  • CI lint/typecheck passes on this branch
  • After merge, next PR's Claude review job logs a ::notice::Resolved latest Opus model: claude-opus-<version> line
  • Review comment posts successfully using the resolved model (same behavior as today, just sourced dynamically)

Made with Cursor

Query the LLM Gateway's /v1/models endpoint at job time and pick the
highest-versioned clean Opus alias (e.g. claude-opus-4-6) rather than
pinning to a single model string. Falls back to claude-opus-4-6 if the
gateway is unreachable or returns no matches, so the workflow stays
functional even when the lookup fails.

This lets the reviewer pick up new Opus families (4-7, 5, etc.) without
a workflow edit, while still allowing a manual bump of the fallback as
a safety net.

Made-with: Cursor
@cb-heimdall
Copy link
Copy Markdown
Collaborator

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

# Match only clean Opus version aliases (e.g. claude-opus-4-6),
# excluding suffixed variants like -default, [1m], #priority.
resolved=$(printf '%s' "$response" \
| jq -r '.data[].id? // empty | select(test("^claude-opus-[0-9]+(-[0-9]+)?$"))' 2>/dev/null \
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex pattern ^claude-opus-[0-9]+(-[0-9]+)?$ only matches single-digit version components. This would fail to match versions like claude-opus-10 or claude-opus-4-10. Consider using [0-9]+ instead of [0-9] to match multi-digit version numbers:

Suggested change
| jq -r '.data[].id? // empty | select(test("^claude-opus-[0-9]+(-[0-9]+)?$"))' 2>/dev/null \
| jq -r '.data[].id? // empty | select(test("^claude-opus-[0-9]+(-[0-9]+)?$"))' 2>/dev/null \

@github-actions
Copy link
Copy Markdown
Contributor

Review Summary

This PR implements dynamic resolution of the latest Opus model for the Claude review workflow, replacing the hardcoded model version. The implementation is generally solid with proper error handling and fallback behavior.

Finding

I identified one issue with the regex pattern used to filter model names - it currently only matches single-digit version components (e.g., claude-opus-4-6) but would fail on future versions with multi-digit components (e.g., claude-opus-4-10 or claude-opus-10). I've posted an inline comment with the fix.

The rest of the implementation looks correct:

  • Proper error handling with set -euo pipefail
  • Secure handling of API credentials via secrets
  • Graceful fallback to hardcoded version on any failure
  • Appropriate logging with GitHub Actions notice/warning annotations
  • Version sorting with sort -V handles semantic versioning correctly

Once the regex issue is addressed, this should be ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants