fix(email): eval-pass follow-ups — ctx-stamped baseline, refresh limit, usage aggregate

[itomek]

Follow-ups from the #1319 consolidated eval pass, cherry-picked onto main after the #2072 train merged (the commits raced the merge). Three small, independent threads:

• Ctx-stamped baseline (feat(email): pin the context-window envelope (16K target / 32K max) as a requirement #1892's last deliverable): baseline_accuracy.json re-recorded on AMD Strix Halo hardware at the pinned 16K window — category_accuracy 0.8094 → 0.796 with the new ctx_size: 16384 stamp. The old number was measured at the silent 64K default; this is the honest pinned-window figure the release gate can now compare against.
• Scorecard-refresh workflow limit 60 → 25 (fix(email): bulk-triage result aggregation overflows the 16K ctx envelope — chunk the batch/result feedback #2087 interim): the eval pass measured the agent-loop's post-tool aggregation turn at ~8K fixed + ~190 tokens/email, so limit 60 at the workflow's pinned 16K is a guaranteed context_length_exceeded. Limit 25 fits (~12.8K); restore a larger limit when fix(email): bulk-triage result aggregation overflows the 16K ctx envelope — chunk the batch/result feedback #2087's chunking lands.
• Triage usage in the aggregate scorecard (feat(email): optimize the context footprint #1891 live gap): per-scenario performance_summary carried tokens_per_triage correctly on real runs (measured: 1,952 tokens per LLM-classified email on the E4B 32K leg) but the shared build_scorecard aggregate's field whitelist silently dropped it. Fixed via the same post-build enrichment pattern ctx_size already uses, with presence + no-fabrication tests.

Test plan

• tests/unit/eval tests/unit/email: 689 passed (single failure is the known pre-existing test_claude_judge machine-.env artifact — passes in CI)
• Full widened suite on the identical pre-merge content: 1819 passed · lint clean · YAML validated
• The baseline number itself is real hardware evidence (299/299 scored at pinned ctx on Strix Halo; run artifacts retained)

[github-actions]

Verdict: Approve ✅

Three small, independent eval-pass follow-ups: an honest baseline re-recorded at the pinned 16K context window, a workflow triage-limit cut from 60 → 25 so it stops overflowing that window, and a fix so the aggregate scorecard stops silently dropping email's per-triage token metrics. Each thread is well-justified, the two code paths are tested for both the present and absent shapes, and the baseline change is real hardware evidence rather than a hand-edit. Nothing blocking.

🔍 Technical details

🟢 Minor / nits

Aggregate means can use mismatched denominators (src/gaia/eval/benchmark.py:453) — _aggregate_triage_usage builds llm_tokens_vals, classified_vals, and per_triage_vals as three independent lists, so if one run had llm_classified_count == 0 (no tokens_per_triage emitted) but a non-zero triage_llm_tokens, the reported tokens_per_triage would be averaged over a different run set than the token/count means it should reconcile with. In practice every run scores the identical corpus so this can't realistically fire — noting it only as a latent inconsistency, not a merge blocker. No change required.

Strengths

• Baseline stamp is generator-produced, not hand-edited — ctx_size: 16384 matches what score_baseline.py:192 emits from recorded_ctx, and the numbers reconcile (238/299 = 0.796), so the "honest pinned-window figure" claim holds up. tolerance_pp: 5 leaves the gate appropriate headroom.
• Both shapes tested — the new aggregate path has a presence test and a no-fabrication absence test (test_benchmark.py:319, :346), mirroring the existing per-run coverage. Good regression pinning for a bug that only showed on real, fully-passing hardware runs.
• Clean reuse of the existing pattern — merging triage usage via post-build_scorecard enrichment follows the same convention ctx_size already uses, rather than reaching into the shared generic scorecard builder. The None-when-absent contract keeps non-email categories untouched.
• Workflow change is defense-in-depth documented — the 60 → 25 cut is justified inline with the token math (~8K fixed + ~190/email vs the 16K window) and both the input default and the LIMIT env fallback were updated together, so a manual workflow_dispatch can't silently reintroduce the overflow.

[itomek]

Acknowledging the denominator note as-is (reviewer marked it non-blocking and it cannot fire while every run scores the identical corpus): the three lists are per-run parallel today, and the case where they'd diverge — a run with classified_count 0 but non-zero triage tokens — would itself indicate broken accounting upstream. Worth revisiting if #2087's chunking changes run composition; noted there implicitly via the shared code path.