Getting ready for release 4.10.1

Author: FrancescAlted

ANNOUNCE.rst

@@ -1,44 +1,37 @@
-Announcing Python-Blosc2 4.10.0
-==============================
-
-This is the string-support release: string expressions and DSL kernels now
-run on miniexpr, ``utf8()`` and ``dictionary()`` columns gain full indexing
-and comparisons, and NumPy's ``StringDType`` is understood by the array
-constructors. Alongside, slicing with plain keys is up to 1.7x faster and a
-new ``blosc2.random`` module brings chunk-parallel, NumPy-quality random
-constructors.
-
-- **String expressions and DSL kernels over strings.** Concatenation,
-  ``lower``/``upper``/``strip``/``replace``/``substr``/``split_part`` and
-  friends now run on miniexpr over fixed-width ``<Un`` and bytes ``S``
-  arrays, producing string results sized by miniexpr itself. ``utf8()``
-  columns can be queried in expression form
-  (``t.where("name == 'x'")``), with scalar comparisons 5-6x faster via a
-  raw-byte scan, and new ``blosc2.utf8_array()`` builds variable-length
-  arrays directly.
-
-- **Full indexing for string columns.** ``create_index()`` now works on
-  ``utf8()`` and ``dictionary()`` columns via alphabetical ranks —
-  ``sort_by`` drops from 424 ms to 7 ms at 1M rows — and scalar comparisons
-  are served from the index.
-
-- **New ``blosc2.random`` module**: 42 of NumPy's 43 ``Generator`` methods,
-  each chunk generated in parallel with its own seeded ``PCG64`` stream
-  (~3x faster than the NumPy path on 100M elements).
-
-- **Slicing up to 1.7x faster.** Plain slice/int keys skip ndindex's general
-  machinery (it was 43% of a scattered-read loop); strided steps, ellipsis
-  and fancy indexing still use it.
-
-- **String plumbing**: ``from_utf8()``/``to_utf8()`` conversions,
-  ``CTable.add_column(values=)``, ``Column.assign()`` on variable-length
-  columns, ``StringDType`` dispatch in the array constructors, and DSL
-  operands can be native NumPy arrays or pandas ``Series``.
-
-- **Important fixes**: string column indexes returning zero rows at the
-  default column width, ``SChunk`` slices for typesizes above 255 bytes
-  (upstream, via C-Blosc2 3.3.1), scalar bools in tuple keys now matching
-  NumPy, and a batch of miniexpr correctness fixes.
+Announcing Python-Blosc2 4.10.1
+===============================
+
+A correctness release: lazy indexing and reductions now follow NumPy in a
+batch of cases where they quietly did not, the stores close several
+cross-process read races, and wheels finally ship usable C-Blosc2 development
+files. Bundled C-Blosc2 moves to 3.3.2.
+
+- **Lazy indexing now matches NumPy.** Integer indexing no longer squeezes
+  length-1 axes the index kept, a ``None`` in the key stops shifting operand
+  axes for ``LazyUDF`` and broadcast operands, indexing a full reduction
+  slices the operands instead of evaluating over everything, and
+  ``datetime64``/``timedelta64`` comparisons work in expressions rather than
+  raising.
+
+- **``NDArray.nbytes`` reports the logical size**, ``size * itemsize``, as
+  NumPy does. ``.schunk.nbytes`` still gives the padded figure, which is what
+  ``cratio`` keeps measuring.
+
+- **``CTable.where()`` applied a short boolean mask to the wrong rows.** A
+  mask no longer than the live-row count is now logical — entry *i* selects
+  the *i*-th live row — instead of being padded out to the physical length
+  and picking up rows outside the view.
+
+- **Cross-process store fixes.** ``EmbedStore`` and ``DictStore`` resolved a
+  key under the store lock but read the data after releasing it; the resolve
+  and the read now share one lock. Overwriting an external ``DictStore`` leaf
+  is atomic too — the new leaf is built beside its final name and moved into
+  place — so a concurrent reader can no longer open a half-rewritten file.
+
+- **Wheels ship working C-Blosc2 development files** (``pkg-config`` and
+  ``find_package(Blosc2)`` both failed against an installed wheel before),
+  and are ~1.5 MB smaller, carrying two copies of ``libblosc2`` rather than
+  three.
 
 Install it with::
 

RELEASE_NOTES.md

@@ -2,7 +2,103 @@
 
 ## Changes from 4.10.0 to 4.10.1
 
-XXX version-specific blurb XXX
+A correctness release: lazy indexing and reductions now follow NumPy in a
+batch of cases where they quietly did not, the stores close several
+cross-process read races, and wheels finally ship usable C-Blosc2 development
+files.  Bundled C-Blosc2 moves to 3.3.2.
+
+### Bug fixes
+
+#### Lazy expressions and indexing
+
+- **Indexing a lazy expression with an integer squeezed too much.**
+  `expr[0]` dropped *every* length-1 axis of the result, including ones the
+  index kept, so a `(1, 4)` expression indexed at `[0]` came back as `(4,)`
+  where NumPy gives `(4,)` only for the consumed axis and keeps the rest.
+  Only the dimensions the integer indices actually consumed are dropped now.
+  `where()` results, whose length is data-dependent, are left alone.
+  Closes #319.
+- **`LazyUDF` and broadcast operands mis-indexed on `None`.** A `None` in the
+  key inserts an axis in the result but consumes none in the operand;
+  aligning it as if it did shifted every axis to its left by one, so
+  `expr[None, 2]` read the wrong operand region. Closes #403, #688.
+- **Indexing a full reduction materialized the whole operand.** `(a + b).sum()
+  [key]` evaluated the reduction over everything and then indexed; the
+  operands are sliced first now. Closes #457.
+- **Datetime comparisons in lazy expressions raised.** numexpr has no datetime
+  type, so `t1 < t2` on `datetime64`/`timedelta64` died with `unknown type
+  datetime64[s]` and the error was re-raised rather than letting the NumPy
+  fallback try. They are compared as their underlying int64 counts, which is
+  exact, keeping the fast path. Mixed units and `NaT` decline that route and
+  fall back to NumPy, since raw counts would silently lie about both.
+  Closes #409.
+- **`NDArray.nbytes` reported the padded size.** It now returns the logical
+  `size * itemsize`, matching NumPy, whenever the shape does not fill the
+  chunk grid exactly. `cratio` still measures the stored (padded) data, so
+  `nbytes / cbytes` need not equal `cratio`; use `.schunk.nbytes` for the
+  padded figure. Closes #544.
+
+#### Tables and stores
+
+- **`CTable.where()` applied a short boolean mask to the wrong rows.** A mask
+  shorter than the live-row count was padded out to the *physical* length,
+  which aligned it with the underlying column and selected rows outside the
+  view. A mask no longer than the live-row count is now treated as logical —
+  entry *i* selects the *i*-th live row — with a short one simply leaving the
+  trailing rows unselected. Closes #607.
+- **Cross-process read races in `EmbedStore` and `DictStore`.** Both resolved
+  a key under the store lock but read the data after releasing it, so a
+  concurrent writer could be caught mid-mutation: `EmbedStore.__getitem__`
+  returned bytes from a stale offset, and `DictStore.__getitem__` opened an
+  external leaf that a concurrent overwrite had just removed or half-rewritten
+  (`KeyError`, or `RuntimeError: Error while getting the buffer`). The
+  resolve, the existence check and the open now share one lock; non-shared
+  stores skip it entirely. Fixes #691, #692.
+- **Overwriting an external `DictStore` leaf is now atomic.** `__setitem__`
+  removed the old leaf and rebuilt it at the same path, and the handle
+  `__getitem__` returns holds no file descriptor — the C layer re-opens the
+  leaf by path for every chunk it decompresses — so a read already in flight
+  could open a truncated file. The new leaf is built beside its final name and
+  moved in with a single `os.replace()`, so every such re-open sees one
+  complete cframe or the other. A crash mid-write now leaves a stray `.tmp`
+  staging file rather than a partial leaf.
+- **`TreeStore.close()` swallowed inline handle failures.** A `CTable` that
+  failed to flush left an archive whose row count disagreed with a varlen
+  column, reported only on read, long after `close()` said it succeeded. Every
+  handle still gets a chance to close and the store is still packed; the
+  failure is then re-raised.
+
+#### Other
+
+- **`unpack_tensor()` turned padding into a phantom column.** `np.dtype()`
+  renames the empty-named padding fields of a structured descr (`''` -> `f2`),
+  so a packed tensor with padding came back with an extra field. Closes #287.
+
+### Packaging
+
+- **Wheels ship usable C-Blosc2 development files.** Install paths are now
+  relative to `CMAKE_INSTALL_PREFIX`; the absolute ones made C-Blosc2 generate
+  a `blosc2.pc` and exported targets pointing into the build tempdir, so
+  `pkg-config` and `find_package(Blosc2)` both failed against an installed
+  wheel. Verified by building and linking a C program against a wheel three
+  ways: `pkg-config`, `Blosc2::blosc2_shared` and `Blosc2::blosc2_static`.
+  miniexpr's license texts are mirrored into `.dist-info/licenses`, where PEP
+  639 tooling looks. Closes #627.
+- **Wheels carry two copies of `libblosc2` instead of three**, ~1.5 MB
+  smaller. C-Blosc2 set both `VERSION` and `SOVERSION`, and scikit-build-core
+  follows symlinks, so the fully versioned file — referenced by nothing but
+  the symlinks pointing at it — was shipped as a third full copy.
+
+### Development
+
+- The test suite runs in parallel by default (`-n auto --dist loadfile` in
+  `pytest.ini`), 130s -> 35s locally, falling back to a serial run when
+  pytest-xdist is absent. The `heavy` tests, 58% of everything collected and
+  excluded from every push-time job, now run in a nightly workflow. The
+  network tests run once per push on a single Linux job instead of five times.
+- The ruff rule set is spelled out with `select` rather than `extend-select`,
+  so a ruff release widening its defaults no longer redefines what CI
+  enforces.
 
 ## Changes from 4.9.1 to 4.10.0
 

pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "rich",
     "threadpoolctl; platform_machine != 'wasm32'",
 ]
-version = "4.10.1.dev0"
+version = "4.10.1"
 [project.entry-points."array_api"]
 blosc2 = "blosc2"
 

src/blosc2/version.py

@@ -1,2 +1,2 @@
-__version__ = "4.10.1.dev0"
+__version__ = "4.10.1"
 __array_api_version__ = "2024.12"