H3 aggregation of categorical_precipitation_type_surface: weights ignored, ties arbitrary, and nulls win the mode

[JackKelly]

Filed by Claude Code, at Jack's direction, from the planning of #496 + #477.

_process_chunk_for_1_lead_time_and_1_ens_member in
packages/dynamical_data/src/dynamical_data/ecmwf_ens/convert_to_polars.py aggregates the numeric
NWP variables to an H3 cell as a proportion-weighted sum, but aggregates
categorical_precipitation_type_surface with:

pl.col(categorical_vars).mode().first(ignore_nulls=True)

That is an unweighted mode. It discards the proportion column entirely, so a grid point
contributing 5% of a cell's area counts exactly as much as one contributing 90%.

The two defects

1. The weights are ignored. A cell fed by three grid points weighted 0.9 / 0.05 / 0.05 returns
the minority category whenever the two slivers happen to agree with each other and disagree with
the dominant point. The cell reports, say, snow because 10% of its area is snowy.

This matters more than the raw geometry suggests: the production H3 grid (H3 resolution 5 against
the 0.25° ECMWF grid, over the GB boundary) averages 2.93 grid points per cell, and cell weights
range down to 0.0204, so lopsided cells are ordinary rather than exotic.

2. .first() on a tie is arbitrary. mode() returns every modal value, in no documented
order, and .first() takes whichever comes out first. With ~3 points per cell, a 1-1-1 or 1-1 tie
is common, so the chosen category is effectively unspecified — and not necessarily stable across
Polars versions.

Why it was not fixed alongside #496

#496 explicitly scopes this variable out ("categorical_precipitation_type_surface is aggregated
by mode().first(ignore_nulls=True), so it already ignores nulls and needs no change"). That
claim is false -- mode() counts null as a candidate value, so a cell whose points are mostly
null comes out null even though real values arrived. See the comment below for the demonstration
and the blast radius; it is defect 3, and it is the most consequential of the three. The weighting
defect is a separate, pre-existing problem that was noticed while confirming the #496 claim.

Sketch of a fix

An area-weighted mode: group the cell's points by category, sum proportion within each category,
and take the category with the largest total weight. Ties then need a deterministic rule (lowest
category index is the obvious one) rather than whatever mode() happens to return.

Worth deciding as part of the issue whether a weighted mode is even the right aggregation for a
categorical precipitation type, or whether the dominant-by-area point's value is a better answer.

Verification

convert_to_polars.py carries a "Convention-sensitive to real ECMWF ENS data" comment, so after
changing it run the network-gated test manually:

uv run pytest --run-network -m network

test_convert_categorical_precipitation_type in
packages/dynamical_data/tests/test_convert_to_polars.py is the existing coverage; it uses five
equally-weighted points, so it cannot see either defect and will need a lopsided-weights sibling.

[JackKelly]

Written by Claude Code, at Jack's direction, after the fourth adversarial review of
PR #516.

Correction: this issue's "Why it was not fixed alongside #496" section was wrong

That section says #496's claim — "categorical_precipitation_type_surface is aggregated by
mode().first(ignore_nulls=True), so it already ignores nulls and needs no change" — is "true of
the null-handling question that issue is about". It is not true, and I have edited the
section to say so. There is a third defect, and it is the most consequential of the three.

Defect 3: nulls compete in the mode, so a cell with data can come out null

mode() counts null as a candidate value. When nulls are the strict mode of a group,
mode() returns [null] and first(ignore_nulls=True) has nothing to fall back on:

>>> import polars as pl; pl.__version__
'1.43.2'
>>> df = pl.DataFrame({"g": [1, 1, 1, 2, 2, 2, 3, 3, 3, 3],
...                    "v": [1, None, None,  None, 7, None,  5, 5, None, None]})
>>> df.group_by("g").agg(pl.col("v").mode().alias("mode_list"),
...                      pl.col("v").mode().first(ignore_nulls=True).alias("agg")).sort("g")
shape: (3, 3)
┌─────┬───────────┬──────┐
│ g   ┆ mode_list ┆ agg  │
╞═════╪═══════════╪══════╡
│ 1   ┆ [null]    ┆ null │   # 2 nulls beat one real value
│ 2   ┆ [null]    ┆ null │   # 2 nulls beat one real value
│ 3   ┆ [null, 5] ┆ 5    │   # a tie, so ignore_nulls rescues it
└─────┴───────────┴──────┘

So ignore_nulls=True only rescues the tie. The real rule is: the cell is null whenever
nulls strictly outnumber every distinct value that did arrive.

Why that matters

Cell sizes on the V1 grid (H3 resolution 5 over the GB boundary, 0.25° ECMWF grid, 1671 cells):

Grid points in the cell	1	2	3	4	5
Number of cells	10	707	343	607	4

• a 3-point cell goes null on two missing points, with a perfectly good third;
• a 4-point cell goes null on two missing points whenever the two survivors disagree — ordinary at
  a precipitation boundary, which is exactly where this variable varies.

And a null here is fatal for any init_time after 2024-11-12, via
Nwp._check_variables_that_were_introduced_after_start_of_dataset. So a couple of adjacent dropped
grid points costs the whole run.

This is not a regression, and PR #516 does not change it

main aggregates this column with the identical expression, so the behaviour is unchanged. It
matters now for two reasons. First, PR #516's headline is that scattered grid-point corruption is
absorbed by a cell's other points — true for the numeric variables, and not true for this one,
so the absorption is narrower than it sounds. Second, this is now the easiest route to a fatal
null in the whole ingest, which makes it the reason a run gets thrown away.

PR #516 corrected the three docs passages that asserted the false universal; it deliberately does
not touch the code, since #496 scoped this variable out and the fix belongs here.

Consequence for the fix

Whatever replaces mode().first(ignore_nulls=True) must decide the null question explicitly rather
than inheriting it. The natural answer, and the one consistent with what PR #516 did for the
numeric variables, is a weight-weighted mode over the non-null points only, with the cell going
null only when no point supplied a category — i.e. argmax over categories of sum(proportion),
restricted to non-null rows, ties broken deterministically (lowest category code). That fixes all
three defects in one expression.

[JackKelly]

Written by Claude Code, at Jack's direction.

Jack asked for this to be fixed in
PR #516 rather than left as
follow-up work, so it is done there and this issue will close when that merges. All three defects
went together, because they were one expression.

What replaced mode().first(ignore_nulls=True)

The proportion weights are summed per category, the heaviest category wins, and an exact tie is
broken on the lowest category code. Points that supplied no category are excluded from the ranking
rather than competing in it, so the cell is null only when no point supplied one — the same rule
the numeric variables follow after #496.

That fixes all three:

• Weights ignored — the ranking is now by area covered, not by number of grid points. A cell
  weighted 0.9 / 0.05 / 0.05 returns the dominant point's category.
• Arbitrary tie — the lowest category code wins. Arbitrary, but deterministic, which is the
  property that was actually missing; the docstring and the known-issues page both say so plainly
  rather than implying the choice is meaningful.
• Nulls winning the vote — missing points are out of the running entirely.

On the tie-break

Worth flagging for review rather than burying: ties resolve toward the lowest code, and code 0
is "no precipitation". So an evenly-split cell leans to the drier category. Any deterministic rule
is equally arbitrary here and exact ties need exactly-equal Float32 weight sums, so this is not a
bias in any practical sense — but if you would rather ties leaned wet, it is a one-word change.

Tests

Four new tests in packages/dynamical_data/tests/test_convert_to_polars.py. Two fail on the
previous implementation:

• test_convert_weights_the_categorical_variable_by_area — fails with the minority category.
• test_convert_does_not_let_missing_points_win_the_categorical_vote — fails with
  ValueError: categorical_precipitation_type_surface must not be null for init_time > 2024-11-12,
  which is exactly the run-discarding path.

The other two are labelled regression guards in their own docstrings rather than claimed as
proof: a tie-break cannot be made to fail reliably against an implementation whose answer was
unspecified rather than wrong, and the all-null-cell case nulled the cell before too, for the wrong
reason.