Roadmap: ensemble-statistics and same-run NWP interpolation experiments

[JackKelly]

Written by Claude Code, at Jack's direction, following the investigation of the failed
2026-08-09 ecmwf_ens partition.

Stacked on #493 — it
branches from that PR and links to two anchors the rewritten known-issues page introduces, so
merge #493 first. The diff shown here is docs-only once #493 lands.

Two new experiments on XGBoost improvements,
both prompted by the 2026-08-09 ingest failure and the discussion it started about how the
pipeline should treat missing NWP.

Tier 2 — fill occasional NWP nulls by interpolating within the same run

Framed as the experiment that tests the page's standing "leave the chronic de-accumulated nulls
un-imputed" position rather than assuming it either way.

The substantive content is why same-run interpolation is the only fill worth trying. Filling
from the previous run is the obvious idea and it is the wrong one: ECMWF regenerates its
perturbations every cycle, so ensemble member 34 of today's 00Z run and member 34 of yesterday's
are not the same trajectory continued — they are two unrelated draws sharing a label, and only
member 0 (the unperturbed control) is comparable across runs. A cross-run fill would graft one
weather scenario's irradiance into another scenario's row, which for solar breaks coherence
between exactly the pair of variables that must agree. Neighbouring steps within a run are the
same trajectory, so that problem disappears — and no row ends up mixing NWP lead times either.

Also records where the fill belongs (the feature engineer, never the ingest, on provenance
grounds), the is_imputed flag that makes a win interpretable, and two caveats: these are rates
over the preceding step, so interpolating across a 6-hourly gap is coarse exactly in the 3–10 day
band users act on; and training on filled inputs reintroduces train/serve skew.

Tier 4 — ensemble statistics (quantiles) instead of member-by-member rows

Placed next to "Train on more ensemble members" (#148),
because the two are alternatives at the same fork rather than sequential steps — worth deciding
together.

Argues for quantiles (p10/p25/p50/p75/p90) rather than mean and spread: mean-and-spread assumes
a near-Gaussian member distribution, which cloud cover badly violates ("half the members overcast,
half clear" is common, consequential, and its mean describes no member at all). Quantiles keep the
marginal shape at a ~5–10× row reduction rather than 51×, which makes it a much smaller bet while
testing the same idea.

Advantages recorded: resilience by construction (a missing member shifts a quantile instead of
removing rows — the 2026-08-09 incident would not have been an event); up to 51× fewer rows, which
eases the memory ceiling, the 32-bit row-index ceiling and the one-laptop principle at V2 scale;
spread as an explicit input, the natural feed for band-widening; and no member-identity problem.

Disadvantages recorded: it gives up the weather-versus-model uncertainty decomposition — with
no per-member conditional distribution there is no linear pool and no source for delivery
Representation 3, so the model cannot answer "is this uncertain because the weather is uncertain,
or because the model is?"; it aggregates in weather space rather than power space, which is the
wrong order of operations for a non-linear response; per-variable quantiles break cross-variable
coherence (a p90 irradiance and p90 temperature need not co-occur in any member); 51× less
training data, in direct tension with #148; and it moves the AllFeatures primary key, so
principle 8 requires a controlled comparison.

One honest counterweight is noted rather than buried: conflating the two uncertainty terms removes
the double-counting risk by construction, so the collapsed model might be better calibrated while
being less informative. Calibration and attribution are different goods.

Evaluation notes that this item specifically needs the probabilistic metrics, not just NMAE —
"matches on NMAE, loses on spread-skill" is the result that would tell you the decomposition was
doing real work.

Verification

pymarkdown scan and mkdocs build --strict pass, and the rendered HTML for the new sections was
read (10 list items, 3 nested lists, all labels intact) — Python-Markdown's list handling is
stricter than the linters catch.

Not included

No GitHub issues created for either experiment yet — say the word and I'll open them against the
right epic with the usual labels/Type/project fields.

🤖 Generated with Claude Code