Merge pull request #694 from Blosc/mask-based-nulls

Mask-based nullable columns for CTable

Author: FrancescAlted

RELEASE_NOTES.md

@@ -1,9 +1,229 @@
 # Release notes
 
-## Changes from 4.10.1 to 4.10.2
+## Changes from 4.10.1 to 4.11.0
 
 XXX version-specific blurb XXX
 
+### New features
+
+#### Mask-based nullable columns for CTable, and they are now the default
+
+A nullable CTable column keeps its nulls in a per-column **validity sidecar** —
+Arrow's own model — instead of reserving a value from its own range. This is now
+what a bare `nullable=True` resolves to, and what every nullable column inferred
+from Arrow, Parquet or CSV gets:
+
+```python
+blosc2.bool(nullable=True)  # no reserved 255; dtype stays np.bool_
+blosc2.int8(nullable=True)  # all 256 values usable, plus nulls
+blosc2.utf8(nullable=True)  # any string, including "" and "\x00"
+blosc2.complex128(nullable=True)  # nullable at all, for the first time
+```
+
+This is what makes nullability **lossless**: a sentinel steals a value from the
+dtype, so a nullable `int8` could not hold `-128`, a free-text `utf8` column had
+no safe sentinel at all, and Arrow columns whose type had no value to spare could
+not be imported. `to_arrow(from_arrow(x))` now returns `x` for nullable `bool`,
+full-range `int8`/`uint8`, `float64` containing `nan`/`±inf`/`-0.0` as values,
+`utf8` containing `""` and `"__BLOSC2_NULL__"`, and `timestamp` with `int64.min`
+as a value — none of which round-trip through a sentinel.
+
+Under mask storage `None` is how you write a null (`t.append((None,))`,
+`t["price"][3] = None`), which a fixed-width sentinel column cannot accept at
+all. `is_null()` is unchanged and remains the uniform API across every kind.
+
+**Nothing on disk changes.** The new default governs *creation* only: opening a
+stored table never re-resolves anything, so every existing table keeps the
+storage, dtype and sentinel it was written with, and every rewrite rule for the
+reserved `255` stays permanently in place. Sentinel storage is supported
+indefinitely and is one keyword away, per column (`null_storage="sentinel"`, or
+any explicit `null_value=`) or globally through `NullPolicy`. Setting a type-wide
+`NullPolicy` sentinel field still implies sentinel storage for the kinds it
+covers, so existing `NullPolicy(float_value=...)` code is unaffected — with one
+unavoidable exception: `255` is the only value a nullable bool may reserve, so it
+is also `bool_value`'s default, and `NullPolicy(bool_value=255)` carries no
+information to act on. A bool column that wants a sentinel has to say so with
+`null_storage` or `column_null_values`.
+
+A table containing a mask column records **schema version 3**. Only such tables do:
+a table with no nullable column still records version 1, exactly as before, and a
+sentinel one does too. Readers older than 4.11.0 refuse a version-3 table rather
+than misreading it, but their message is a bare `ValueError: Unsupported schema
+version 3` — the hint naming `convert_nulls(to='sentinel')` ships in 4.11.0, so
+only readers that can already open the file will print it.
+
+If some of your data has to stay readable by an earlier release, pin the storage
+rather than discovering this downstream. Per column with
+`null_storage="sentinel"` (or any explicit `null_value=`), or process-wide,
+including for schemas inferred from Arrow, Parquet and CSV:
+
+```python
+with blosc2.null_policy(blosc2.NullPolicy(null_storage="sentinel")):
+    t = blosc2.CTable.from_parquet("data.parquet")
+```
+
+That reinstates the sentinel's lossiness — a float column's nulls become `NaN`
+again, and a type with no value to spare still cannot be imported — which is the
+trade being made.
+
+`Column.null_storage` reports where a column keeps its nulls and `info` tags each
+column (`int64 nullable[mask]`), so `CTable.convert_nulls()` can move columns
+between the two in either direction — never implicitly, and refusing rather than
+silently relabelling data when a sentinel is unavailable.
+
+One deliberate semantic difference: in a mask column `NaN` is a **value**,
+following Arrow, and only the sidecar marks a null. Sentinel float columns keep
+NaN-as-null. See "Where nulls are stored" in the CTable reference.
+
+#### Column indexes are null-aware
+
+Every index kind stores per-segment `min`/`max`, and those extrema are now taken
+over the rows that carry a **value**: a column's nulls are read from its
+validity channel and left out, and a segment with no value at all is flagged
+rather than summarised. This applies to both storages — an `INT64_MIN` sentinel
+is exactly as invisible to a summary as a mask column's fill.
+
+Two things follow. `Column.min`/`Column.max` answer from the index for a
+nullable column (**236x** on a 20M-row `int64`, measured) where before every
+nullable column but a NaN-sentinel float had to scan; and `where()` with an `OR`
+over a nullable indexed column uses the index instead of falling back to a full
+scan (**1.6x** on a 20M-row two-column probe). The `OR` fallback existed because
+the only null filtering available was global, and a global filter drops a row
+that is null in one branch but matches the other; the segment path never needed
+it, because it *evaluates* the predicate, which has been null-aware per leaf
+since the string-predicate fix below.
+
+Indexes written by an earlier release are read as not null-aware and keep the
+old fallback, so nothing silently changes meaning; `rebuild_index()` promotes
+them. Building an index over a nullable column that actually holds nulls now
+costs one decompression pass (33 ms for a 20M-row `int64` column) because the
+incremental per-block summaries folded during writes carry no validity; a
+nullable column with no nulls keeps that fast path untouched.
+
+#### Predicates over nulls follow three-valued (Kleene) logic
+
+A comparison against a null is neither true nor false. It is now **unknown**,
+the third value SQL and Arrow both use, and `&`, `|`, `^` and `~` combine it by
+Kleene's rules instead of collapsing it to `False` at the leaf:
+
+```python
+t.where(t.price > 10)  # rows definitely above 10
+t.where(~(t.price > 10))  # rows definitely *not* above 10 — nulls in neither
+```
+
+`where()` keeps what a predicate is **true** for, so the rows it returns for a
+plain comparison are unchanged. What this fixes is everything built on top of
+one. `~(t.price > 10)` used to invert a null that had already been collapsed to
+`False` and so returned every null row — the exact opposite of the intent — and
+`~((a > 10) & (b == 999))` dropped rows that qualify, because `unknown & false`
+is *false*, not unknown, and only a real third value can express that. Both
+query forms are covered, and both now agree with SQL: the string form carries
+the second channel through an AST rewrite under negation.
+
+A predicate can be asked about its unknown rows rather than only filtered with:
+
+```python
+p = t.price > 10
+p.is_null()  # boolean array: the rows it cannot answer for
+p.null_count()
+t.where(p.fillna(True))  # keep what cannot be ruled out
+```
+
+`fillna(False)` is the other reading, and is what `where()` applies implicitly.
+
+Predicates over non-nullable columns are untouched and cost nothing extra; the
+result of a nullable comparison is still a `blosc2.LazyExpr`, so it computes,
+indexes and plans exactly as before. Measured cost of the exact answer: a
+negated two-column conjunction over a 20M-row nullable table runs 1.15x slower
+than the wrong answer it replaces; every other predicate shape is unchanged.
+
+Two consequences worth knowing. `t.where(dict_col != "x")` no longer returns the
+rows where `dict_col` is null (its reserved code differs from every value's, so
+it used to match); `dict_col == None` remains how to ask for them. And
+`Column.isin()` stays deliberately two-valued — it returns a materialized array
+and has its own spelling for nulls (`None` among the values).
+
+### Bug fixes
+
+- **`group_by` returned the wrong `min` for a `bool` value column.** The
+  per-group accumulator was seeded from the dtype's opposite identity, and `bool`
+  had none, so an all-`True` group reduced to `False`. Reachable with any plain
+  non-nullable bool column on the generic aggregation path.
+- **Descending `sort_by` on a `bool` column raised**, and on a signed-integer
+  column holding its dtype's minimum (`-128` for `int8`) that row sorted as if it
+  were the largest. The descending key negated in the column's own dtype, where
+  `bool` has no unary minus and a narrow signed type wraps.
+- **`add_column()` after `copy()` backfilled one row short**, and raised for a
+  variable-length column: the copy recorded its write watermark one below the
+  convention every other writer follows.
+- **A string predicate over a nullable column returned its nulls as matches.**
+  `t.where("a > 10")` compared the stored sentinel, so any sentinel satisfying
+  the predicate (`null_value=999` against `> 10`) came back as a match. The
+  operator form (`t.where(t.a > 10)`) was always correct. Fixed for both storages.
+- **A nullable `uint8` ndarray column came back as `bool`.** The `bool → uint8`
+  widening that sentinel storage needs was undone by dtype rather than by
+  whether it had been applied, so a column declared `uint8` was truncated to
+  flags.
+- **`~` on a nullable bool column selected its nulls.** SQL `WHERE` semantics
+  say a null satisfies neither a predicate nor its negation; the mask path
+  inverted the stored `False` fill instead. (The sentinel path was already
+  correct, via its `== 0` rewrite.)
+- **CSV import and export ignored a validity sidecar.** `to_csv` compared
+  against the sentinel to find nulls, so a mask column wrote its fill as if it
+  were data, and `from_csv` had nothing to put in an empty field and raised.
+  Both go through the sidecar now: an empty CSV field is a null in either
+  direction. Sentinel columns keep writing their sentinel, unchanged.
+- **Reductions on a sorted view of a mask column read the wrong rows.** The
+  null flags were gathered in the view's order and the values in physical
+  order, so `sum()` on a sorted view could return `NaN` from a column whose
+  nulls are not `NaN`, and `unique()` could report the fill as data while
+  dropping a real value. Sentinel columns were unaffected.
+- **`convert_nulls()` flattened a nested column**, even when it had nothing to
+  convert: the schema copy it makes dropped both the table metadata and the
+  logical parent of a nested group, so a struct column came back as its leaves.
+  An in-place conversion on a persistent table wrote that flattened schema to
+  disk. Saving and reopening a nested table dropped the same parent, which is
+  fixed alongside it.
+- **Descending `sort_by` mis-ordered the widest integers.** The key was built
+  by negating, and negation has a fixed point: `int64`'s minimum sorted as if
+  it were the largest, and a `uint64` above 2**63 wrapped negative and sorted
+  below small values.
+- **CSV was written and read in the platform's locale encoding**, so text a
+  column can hold but cp1252 cannot encode — anything outside Latin-1 — raised
+  `UnicodeEncodeError` on Windows. Both directions are UTF-8 now, and reading
+  absorbs a byte-order mark if one is present.
+- **A timestamp column could not be written through `col[key] = value`.** A
+  `datetime` was never encoded to the stored `int64`, so every key form failed;
+  `extend()` was unaffected. ISO strings and `datetime64` are accepted too.
+- **Assigning one value to many rows raised on a mask-storage column.**
+  `col[0:2] = 7` — and the same write through a boolean mask or an index list —
+  failed with `TypeError: iteration over a 0-d array`, because the write path
+  looked for nulls *inside* a value that was a single cell rather than a batch.
+  Scalar broadcast works again, and `col[0:2] = None` now makes every selected
+  row null.
+- **`extend()` from another table lost nulls between storages.** Copying rows
+  from a mask-backed column into a sentinel-backed one (or the reverse, or
+  between two sentinels reserving different values) wrote whatever stood in for
+  the null as real data. Nullity is translated now.
+- **A row read showed a mask column's fill instead of `None`.** `t[i]`,
+  iteration and `repr` surfaced the placeholder that occupies a null slot, which
+  is not part of the format contract — and disagreed with a `vlstring` column in
+  the same row, which already read `None`. Sentinel columns still show their
+  sentinel, which is the value you chose.
+- **`to_numpy(masked=True)` and `dropna()` raised on a nullable dictionary
+  column**, which reported its nulls per physical slot rather than per live row.
+- **A null in a complex column reached pandas as `nan+0j`** rather than as
+  missing.
+- **A nullable `ndarray` of `bool` forgot it had been widened when reopened**, so
+  converting it to mask storage left it `uint8`, and the guard against a
+  dtype-changing in-place conversion on a persistent table stopped firing.
+- **`convert_nulls(to="sentinel")` refused over a value in a deleted row.** The
+  collision check scanned physical slots, so a proposed sentinel present only in
+  a row already deleted — unreadable, and dropped by the next `compact()` —
+  blocked the conversion. Only live rows are consulted now, and the row named in
+  the refusal is the logical one the caller can index rather than a physical
+  slot.
+
 
 ## Changes from 4.10.0 to 4.10.1