ci(email): gate releases on the full eval suite + all email tests

[kovtcharov-amd]

Why this matters

An email-agent release could previously ship even if the judge-based evals regressed, the perf bars were missed, or the unit/spec tests were red. #1983 wired the eval suite (triage / drafting / action-item / briefing) into the release as a gate, but the drafting + perf gates were report-only, and the release did not run the unit or spec/contract tests at all.

After this change, a release of @amd-gaia/agent-email is blocked unless, at the release commit:

• Every eval passes — triage acceptance bar, drafting approval, perf bars, plus the already-enforcing action-item + briefing judge evals.
• A missing ANTHROPIC_API_KEY blocks — the action-item + briefing evals hard-require the judge and fail loudly (no fuzzy-only fallback), so a release can't ship an un-judged suite.
• All email tests pass, including the REST-sidecar contract and OpenAPI-conformance (spec) tests — now part of the release gate, not just PR CI.
• The judge-scored drafting result is folded into SCORECARD.md (draft_approval_rate, reported metric).

What changed

• Enforce drafting + perf gates (tests/fixtures/email/{drafting,perf}_gate_thresholds.json → enforce: true). ⚠️ The perf bars are not yet independently hardware-confirmed — if the stx pool proves noisy, widen them in the manifest (data, not code) rather than reverting to report mode.
• release_agent_email.yml: publish now also needs a new email-tests gate (uses: test_email_agent_unit.yml).
• test_email_agent_unit.yml: installs the email hub package and also runs hub/agents/python/email/tests/ (REST contract) + tests/test_email_openapi_conformance.py (spec) — fully mocked, no Lemonade.
• gen_scorecard.py: new --drafting-report; folds draft_approval_rate as a reported metric (weight 0) so the aggregate stays 100 × within_one_bucket_accuracy. Loud-skip report → metric omitted; malformed non-skip → hard error.
• email_scorecard_refresh.yml: runs the drafting eval and passes --drafting-report, so the committed/shipped scorecard carries the drafting metric.
• test_email_agent_eval.yml: nightly → weekly (Mondays 07:17 UTC) — the run now spends Claude judge tokens, and the release re-runs the suite as a hard gate.

Design note (deviation from the original plan preview)

The approved preview envisioned a bespoke eval-gate job that regenerates + ships a runner-fresh scorecard. #1983 had already landed a reusable email-eval gate, so this adapts to that structure instead: the eval runs (and now enforces) via email-eval; the committed scorecard is kept honest — and gains the drafting metric — by email_scorecard_refresh.yml. Net effect matches the requirement.

Test plan

• pytest tests/unit/eval/test_release_scorecard.py — 76 pass (3 new: drafting metric folded/reported/aggregate-unchanged; loud-skip omits; malformed fails loud). Windows: PYTHONUTF8=1.
• black --check / isort --check clean on the changed Python.
• All four edited workflows YAML-parse; both manifests are valid JSON with enforce: true.
• CI-only (cannot run locally): the email-eval gate + email_scorecard_refresh run on the self-hosted stx pool with Lemonade + ANTHROPIC_API_KEY. Verify via a workflow_dispatch dry-run on hardware before cutting a real tag — especially that the newly-enforced perf bars pass on the pool.

Dependency

Overlaps PR #1989 (docs) on gen_scorecard.py. Trivial to resolve; recommend landing #1989 first.

[github-actions]

Verdict: Approve with suggestions.

This PR turns the email-agent release into a real gate: the drafting + perf eval bars now block (not just report), the release also runs the full unit/contract/spec test suite, and the judge-scored drafting result is folded into the scorecard as a reported metric. The gating plumbing is sound — I verified the enforce: true flips actually flow through eval_gate_report.py (perf) and eval_drafting_report.py (drafting), the report path is consistent (eval-out/drafting_gate_report.json), and the new scorecard logic is well-tested (fold / loud-skip / malformed-fails-loud).

One thing to fix before merge: the eval workflow that is the release gate still describes itself as "report mode / enforce:false" in several comments and echo lines — flipping the manifests made those descriptions contradict the new behavior. Not a code bug, but a maintainer reading that file will be told the gate is non-blocking when it now blocks. Worth correcting in the same PR so the gate doesn't ship self-contradicting docs.

🔍 Technical details

🟡 Important — stale "report mode" text now contradicts the enforce flip

test_email_agent_eval.yml is the workflow the release calls (email-eval), and it still documents the old posture in three places after this PR flips perf + drafting to enforce: true:

• Header, .github/workflows/test_email_agent_eval.yml:25-32 — "All three manifests ship enforce: false, so should_fail is always false today … the build stays green." Now false for perf and drafting.
• Header, :20-21 — lists "the FP/FN, perf, and drafting gates" as report-mode in contrast to the enforcing briefing gate; perf + drafting are now enforcing too (only FP/FN remains report).
• Header, :41-42 — "drafting_gate_thresholds.json, enforce:false) … report mode".
• Step name, :211 — "Voice-drafting quality eval (judge-scored, report mode)".
• Eval summary echo, :315-316 — "Report mode: while the manifests ship enforce:false, a breach … does NOT fail the build."

CLAUDE.md's "a functional change must update every doc that describes it" applies here — the switch changed, its descriptions didn't. Recommend a sweep of that file so its self-description matches the manifests it reads.

🟢 Minor — the new release_agent_email.yml comment mis-attributes the key check

.github/workflows/release_agent_email.yml:39-44 says the missing-ANTHROPIC_API_KEY failure comes from the "voice-drafting eval" ("the eval-gate asserts it and FAILS … so a release never ships an un-judged drafting eval"). But eval_drafting_report.py:56-75 loud-skips and returns 0 when the key is absent — it does not fail. The eval that hard-requires the key is the briefing (and action-item) eval. Net effect is correct (the release is still blocked when the key is missing), but the reasoning points at the wrong step; if someone later disables the briefing gate trusting this comment, un-judged releases would slip through. Suggest re-attributing the assertion to the briefing/action-item eval.

Strengths

• Single-source enforce switch respected — gating is toggled purely in the JSON manifests (data), no workflow edits needed; the perf comment even pre-plans "widen the bars in the manifest if the pool is noisy, don't revert to report mode."
• No silent fallback — _load_draft_approval_rate returns None on a loud-skip (metric omitted, never invented) and raises ValueError on a malformed-but-non-skip report. Matches GAIA's fail-loudly rule.
• Solid test coverage — the three new tests cover fold-as-reported-metric (with an aggregate-unchanged assertion), loud-skip-omits, and malformed-fails-loud — exactly the branches that matter.

[github-actions]

🟡 The newly added comment in the release workflow makes a false claim about how a missing ANTHROPIC_API_KEY is handled — the actual behavior is a silent skip, not a build failure. This means a release could ship without any judged drafting eval if the secret isn't configured.

The comment (added in this PR) says:

REQUIRED: the eval-gate asserts it and FAILS the release if absent, so a release never ships an un-judged drafting eval.

But eval_drafting_report.py does the opposite when the key is absent: it writes {"skipped": True, "should_fail": False} and returns 0 (success). The enforce:true flip only causes a failure when the judge actually runs and finds a breach — absence of the key is always a pass, regardless of the enforce flag.

The comment is incorrect and may mislead maintainers into thinking the gate protects against an un-judged release when it doesn't.

🔍 Technical details

eval_drafting_report.py:56–75:

if not os.environ.get("ANTHROPIC_API_KEY"):
    report = {"skipped": True, "should_fail": False, ...}
    (out / "drafting_gate_report.json").write_text(...)
    return 0   # ← always 0; the eval-gate step passes

The guard at line 127 (if gate.get("should_fail"): return 1) is unreachable when the key is absent because main() returns early at line 75.

Fix: either (a) correct the comment to say "absent → loud skip (gate not enforced)" — matching the actual behaviour and the scorecard-refresh comment — or (b) if the intent really is to require the key for a release, add a check: when thresholds.enforce is true and the key is absent, return 1 instead of 0.

Affected line: .github/workflows/release_agent_email.yml, the newly added ANTHROPIC_API_KEY comment block (lines ~52–58 of the diff).

[github-actions]

🟡 The release workflow's claim that a missing ANTHROPIC_API_KEY fails the release is wrong — eval_drafting_report.py explicitly exits 0 on a missing key and writes a skip report instead. A release can therefore ship without any judge-scored drafting metric if the secret isn't configured, silently bypassing the gate the PR is claiming to add. The enforce: true flip only blocks runs where the key is present and the approval rate drops below 70%; it provides no protection against the key simply being absent.

Additionally, eval_drafting_report.py's module docstring (line 10) still reads (enforce:false) after this PR flipped the manifest to enforce: true. The docstring also instructs maintainers to flip enforce in the manifest to make the gate block — which has now happened, but the docstring doesn't say so.

🔍 Technical details

False enforcement claim — release_agent_email.yml:

# ANTHROPIC_API_KEY — ... REQUIRED: the eval-gate asserts
#                     it and FAILS the release if absent

Conflicts directly with eval_drafting_report.py:15-16:

When ``ANTHROPIC_API_KEY`` is absent … writes a LOUD, explicit skip report
(never an invented pass) and exits 0.

A skip-report causes gen_scorecard.py's _load_draft_approval_rate to return None, which simply omits the metric — it does not block the release. The comment in release_agent_email.yml should either be corrected to match the actual behaviour (skip → omit metric, not fail), or the script needs a sys.exit(1) when thresholds.enforce is True and the key is absent.

Stale docstring — eval_drafting_report.py:10:

# Before (still present):
``tests/fixtures/email/drafting_gate_thresholds.json`` (enforce:false)

Should read (enforce:true) after this PR's manifest change.

[kovtcharov]

Addressed the doc-accuracy review from the enforce:false→true flip — since this PR had already merged, the corrections landed in follow-up #2023 (branch fix/email-eval-fail-loudly):

• Swept the stale report-mode text in test_email_agent_eval.yml (header, gate list, summary echo, drafting/action-item step comments) and the eval_drafting_report.py docstring to match the shipped state: perf + drafting + briefing enforce; only the FP/FN quality gate stays report-mode.
• Corrected the fail-loudly story instead of merely re-describing it. The ANTHROPIC_API_KEY handling was a genuine "No Silent Fallbacks" violation: the drafting eval loud-skipped and returned 0 on a missing key, so a keyless run shipped an un-judged gate while the manifest said enforce:true. It now hard-fails (exit 1) with an actionable error, gen_scorecard.py no longer silently omits the metric, and the release_agent_email.yml comment is corrected to attribute the keyless-release block to the drafting eval's exit 1 (not a skip).

Verified none of the three workflows running the judged eval trigger on fork PRs, so the key is always present in a legitimate run and the hard-fail can't spuriously break nightly/forks. Tests + lint green on top of current main (which now also carries #1989).