ci(email-eval): run eval workflows on the stx pool with version-checked Lemonade#1983
Merged
Conversation
added 3 commits
July 9, 2026 11:13
…esult handling) Adopts the token-reduction and data-out-of-context outcomes from Anthropic's "code execution with MCP" writeup using GAIA's existing embedding-based tool loader and tool-wrapper layers — not the code-execution mechanism, which is a poor fit for local 4B-35B models and a large local-machine attack surface. Scopes three workstreams: WS1 routes MCP tools through the dynamic loader (progressive disclosure, with an MCP-only gating mode so static-tool recall is preserved), WS2 keeps large tool results out of context via a session-scoped artifact store + handle resolution in _execute_tool, WS3 (optional) adds PII pass-through. Non-goal: model-authored code execution / sandbox.
…ed Lemonade The two email-agent eval workflows targeted `runs-on: [self-hosted, lemonade-eval]` — a runner label no machine in the pool advertises — so every run dead-queued indefinitely and neither has ever executed. They were also authored for a Linux box (bash, `venv/bin/activate`, `python <<'PY'` heredocs) and assumed a Lemonade server was already running. Point both at the Windows `stx` pool and adopt the proven test_agent_sdk.yml setup: setup-venv, the install-lemonade action (pins the runner to the expected LEMONADE_VERSION — installs if missing, reconciles if drifted, so new machines are covered), and ensure-lemonade-running.ps1 to start + warm the server. The `lemonade-eval` concurrency group is kept for serial execution (CLAUDE.md). Eval bodies are converted bash -> PowerShell. The large inline gate-reader and voice-drafting heredocs don't survive the shell change, so they're extracted verbatim into eval_gate_report.py / eval_drafting_report.py next to the existing gen_scorecard.py. LEMONADE_BASE_URL uses the bare host so gen_scorecard's appended /api/v1/health path doesn't double.
github-actions
Bot
added
documentation
Contributor
|
Verdict: Approve ✅ This revives two email-agent eval workflows that were silently dead since day one — they targeted a The one thing left is the item the author already flagged as unchecked: the PowerShell conversions couldn't be exercised locally, so a real Windows 🔍 Technical detailsVerified correct
🟢 Minor (non-blocking)
Strengths
|
itomek
previously approved these changes
Jul 10, 2026
Resolve the test_email_agent_eval.yml conflict from main's two new eval blocks (action-item extraction #1964, daily-briefing summary #1951) and the reusable workflow_call trigger, keeping this branch's Windows stx + PowerShell + extracted-script conversion. - Convert both new eval blocks from bash heredocs to the branch's pattern: new packaging/eval_action_item_report.py and eval_briefing_report.py, called from PowerShell steps (matching eval_gate_report.py / eval_drafting_report.py) - Point the top-level env at inputs.* (not github.event.inputs.*) so model/limit propagate for the new workflow_call path as well as workflow_dispatch - Keep per-step env minimal (ANTHROPIC_API_KEY only); the centralized top-level env supplies EMAIL_EVAL_MODEL/LIMIT/EXPERIMENTS
kovtcharov
approved these changes
Jul 10, 2026
4 tasks
kovtcharov-amd
added a commit
that referenced
this pull request
Jul 10, 2026
) ## Why this matters An email-agent release could previously ship even if the judge-based evals regressed, the perf bars were missed, or the unit/spec tests were red. #1983 wired the eval suite (triage / drafting / action-item / briefing) into the release as a gate, but the drafting + perf gates were **report-only**, and the release did **not** run the unit or spec/contract tests at all. After this change, a release of `@amd-gaia/agent-email` is blocked unless, at the release commit: - **Every eval passes** — triage acceptance bar, **drafting** approval, **perf** bars, plus the already-enforcing action-item + briefing judge evals. - **A missing `ANTHROPIC_API_KEY` blocks** — the action-item + briefing evals hard-require the judge and fail loudly (no fuzzy-only fallback), so a release can't ship an un-judged suite. - **All email tests pass**, including the **REST-sidecar contract** and **OpenAPI-conformance (spec)** tests — now part of the release gate, not just PR CI. - The **judge-scored drafting result is folded into `SCORECARD.md`** (`draft_approval_rate`, reported metric). ## What changed - **Enforce drafting + perf gates** (`tests/fixtures/email/{drafting,perf}_gate_thresholds.json` → `enforce: true`).⚠️ The perf bars are not yet independently hardware-confirmed — if the stx pool proves noisy, widen them in the manifest (data, not code) rather than reverting to report mode. - **`release_agent_email.yml`**: `publish` now also needs a new `email-tests` gate (`uses: test_email_agent_unit.yml`). - **`test_email_agent_unit.yml`**: installs the email hub package and also runs `hub/agents/python/email/tests/` (REST contract) + `tests/test_email_openapi_conformance.py` (spec) — fully mocked, no Lemonade. - **`gen_scorecard.py`**: new `--drafting-report`; folds `draft_approval_rate` as a reported metric (weight 0) so the aggregate stays `100 × within_one_bucket_accuracy`. Loud-skip report → metric omitted; malformed non-skip → hard error. - **`email_scorecard_refresh.yml`**: runs the drafting eval and passes `--drafting-report`, so the committed/shipped scorecard carries the drafting metric. - **`test_email_agent_eval.yml`**: nightly → **weekly** (Mondays 07:17 UTC) — the run now spends Claude judge tokens, and the release re-runs the suite as a hard gate. ## Design note (deviation from the original plan preview) The approved preview envisioned a bespoke `eval-gate` job that regenerates + ships a runner-fresh scorecard. #1983 had already landed a reusable `email-eval` gate, so this adapts to that structure instead: the eval runs (and now enforces) via `email-eval`; the committed scorecard is kept honest — and gains the drafting metric — by `email_scorecard_refresh.yml`. Net effect matches the requirement. ## Test plan - [x] `pytest tests/unit/eval/test_release_scorecard.py` — 76 pass (3 new: drafting metric folded/reported/aggregate-unchanged; loud-skip omits; malformed fails loud). *Windows: `PYTHONUTF8=1`.* - [x] `black --check` / `isort --check` clean on the changed Python. - [x] All four edited workflows YAML-parse; both manifests are valid JSON with `enforce: true`. - [ ] **CI-only (cannot run locally):** the `email-eval` gate + `email_scorecard_refresh` run on the self-hosted **stx** pool with Lemonade + `ANTHROPIC_API_KEY`. Verify via a `workflow_dispatch` dry-run on hardware before cutting a real tag — especially that the newly-enforced **perf** bars pass on the pool. ## Dependency Overlaps PR #1989 (docs) on `gen_scorecard.py`. Trivial to resolve; recommend landing #1989 first. --------- Co-authored-by: Ovtcharov <kovtchar@amd.com> Co-authored-by: Kalin Ovtcharov <kalin@extropolis.ai>
This was referenced Jul 13, 2026
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
Jul 13, 2026
…ontract (amd#2042) Closes amd#1994 ## Why this matters The four eval CI gate scripts added in amd#1983 compute the `should_fail` → `return 1` decision that blocks (or fails to block) an email-agent release — and shipped with zero unit tests, so a wrong dict key or truthiness slip would silently neuter the release gate. This PR locks that contract with 22 fully-mocked unit tests (no Lemonade, no Claude, no mailbox), including the one asymmetry a naive test would get wrong: `briefing_gate` is the enforcing gate, so a judge/generation outage on a skip must **fail** the build (`should_fail` mirrors `enforce`), while the three report-mode siblings return 0. Test-only change — no production file is touched. ## Evidence (CLI) Full email test suite in a fresh worktree venv — the 22 new tests collect and pass alongside all siblings: ``` $ .venv-wt/bin/python -m pytest hub/agents/python/email/tests/ -v --tb=short ... test_eval_briefing_report.py::test_skipped_gate_variant_blocks_the_build PASSED test_eval_gate_report.py::test_default_limit_and_experiments_passed_to_run_benchmark PASSED ======================== 340 passed, 1 warning in 2.91s ======================== ``` ``` $ python util/lint.py --all [SUCCESS] ALL QUALITY CHECKS PASSED! ``` <details> <summary>🔍 Coverage detail per script</summary> Per script (`hub/agents/python/email/packaging/eval_{gate,drafting,briefing,action_item}_report.py`): - `should_fail` → exit-code contract: breach → 1; pass / missing key / report-mode skip → 0; **briefing enforcing skip → 1** (`briefing_quality.py:434-442`). - Fail-loud `ANTHROPIC_API_KEY`-absent path (3 judge-scored scripts): returns 1 with every generation/judge/corpus-loader fake asserted **never called** and no report JSON written. - Report JSON emission: path + top-level keys per script (incl. `match_mode`, unique to action_item). - `EMAIL_EVAL_LIMIT`/`EMAIL_EVAL_EXPERIMENTS` defaults (50/1) asserted via a capturing `run_benchmark` fake. - Mock hygiene: patched on the loaded module object (scripts bind `gaia.eval` names at load), shapes transcribed from the real producers with `# shape from src/gaia/eval/...` comments, happy-path tests assert fakes WERE called, autouse `chdir(tmp_path)` + sentinel-API-key fixtures keep tests hermetic. </details> ## Test plan - [x] `.venv-wt/bin/python -m pytest hub/agents/python/email/tests/ -v --tb=short` — 340 passed (22 new) - [x] `python util/lint.py --all` — all gates pass - [x] `test_email_agent_unit.yml` green on this PR (`ready_for_ci` label applied — the workflow skips drafts without it) --------- Co-authored-by: Tomasz Iniewicz <tomasz@iniewicz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this matters
Both email-agent eval workflows have been silently dead for their entire existence: they targeted
runs-on: [self-hosted, lemonade-eval], a runner label no machine in the pool advertises, so every trigger queued forever and neither has ever executed a single time. On top of that they were written for a Linux box (bash,venv/bin/activate,python <<'PY'heredocs) and assumed a Lemonade server was already up — none of which holds on our actual self-hosted fleet.After this change they run on the Windows
stxpool and, critically, provision their own Lemonade: theinstall-lemonadeaction pins each runner to the expectedLEMONADE_VERSION(installs if missing, reconciles up/down if drifted), so a brand-newstxmachine works with no manual setup. This mirrors the proventest_agent_sdk.ymlpattern already used by ~12 workflows.Scope is the two email evals only.
test_eval_rag.ymlhas the same dead label but is a separate rebuild tracked in #1315 (its header lists further blockers + a 90-min overflow), so it's deliberately left alone.What changed
runs-on→ the standardstx/stx-testpool togglesetup-venv+install-lemonade+ensure-lemonade-running.ps1(start & warm the version-matched server); kept thelemonade-evalconcurrency group for serial eval execution (CLAUDE.md)eval_gate_report.py/eval_drafting_report.pynext to the existinggen_scorecard.pyLEMONADE_BASE_URLset to the bare host sogen_scorecard's appended/api/v1/healthdoesn't double to/api/v1/api/v1Test plan
black,isort,flake8(repo CI config) clean on both new scripts;py_compilepassesgaia.eval.benchmark/gaia.eval.draft_qualitystxrunner — the PowerShell conversions can only be confirmed on the actual hardware (couldn't be exercised locally). Run Email Agent Eval — scorecard refresh and Email Agent Eval (offline, report mode) via Run workflow and confirm they reach + complete the benchmark.