|
1 | | -Announcing Python-Blosc2 4.9.1 |
| 1 | +Announcing Python-Blosc2 4.10.0 |
2 | 2 | ============================== |
3 | 3 |
|
4 | | -This is a hot-fix release for the Arrow interop work introduced in 4.9.0 — |
5 | | -one real performance regression and one clearer error message, both in |
6 | | -``CTable``. |
7 | | - |
8 | | -- **Faster dictionary-column Arrow export**: ``CTable.iter_arrow_batches()`` |
9 | | - (and therefore ``to_arrow()`` and the Arrow PyCapsule interchange, |
10 | | - ``__arrow_c_stream__``) was recomputing the full live-row-position array |
11 | | - from scratch on every batch, for every dictionary-encoded string column — |
12 | | - an ``O(n_rows)`` scan repeated ``O(n_rows / batch_size)`` times. It's now |
13 | | - computed once per export call instead. 6-14x faster export for |
14 | | - dictionary columns on a 1M-row benchmark. |
15 | | - |
16 | | -- **Worth knowing regardless of this fix**: the Arrow PyCapsule protocol |
17 | | - (``__arrow_c_stream__``) has no column-projection pushdown — a consumer |
18 | | - that only needs two columns (DuckDB, pyarrow, Polars, pandas) still |
19 | | - triggers export of *every* column in the table, since the raw Arrow C |
20 | | - Stream interface has no way to say "I only need these." Use |
21 | | - ``CTable.select([...])`` to project down to the columns you actually |
22 | | - need before handing the table off, especially if any column holds an |
23 | | - expensive nested/list type:: |
24 | | - |
25 | | - sub = t.select(["company", "fare"]) |
26 | | - duckdb.sql("SELECT company, avg(sub.fare) FROM sub GROUP BY company").show() |
27 | | - |
28 | | -- **Clearer error on ``mode="a"``**: opening a ``CTable`` with |
29 | | - ``mode="a"`` at a path that doesn't exist yet now raises a |
30 | | - ``FileNotFoundError`` explaining that ``mode="a"`` opens an existing |
31 | | - table (use ``mode="w"`` to create one), instead of silently creating a |
32 | | - new, empty table. |
| 4 | +This is the string-support release: string expressions and DSL kernels now |
| 5 | +run on miniexpr, ``utf8()`` and ``dictionary()`` columns gain full indexing |
| 6 | +and comparisons, and NumPy's ``StringDType`` is understood by the array |
| 7 | +constructors. Alongside, slicing with plain keys is up to 1.7x faster and a |
| 8 | +new ``blosc2.random`` module brings chunk-parallel, NumPy-quality random |
| 9 | +constructors. |
| 10 | + |
| 11 | +- **String expressions and DSL kernels over strings.** Concatenation, |
| 12 | + ``lower``/``upper``/``strip``/``replace``/``substr``/``split_part`` and |
| 13 | + friends now run on miniexpr over fixed-width ``<Un`` and bytes ``S`` |
| 14 | + arrays, producing string results sized by miniexpr itself. ``utf8()`` |
| 15 | + columns can be queried in expression form |
| 16 | + (``t.where("name == 'x'")``), with scalar comparisons 5-6x faster via a |
| 17 | + raw-byte scan, and new ``blosc2.utf8_array()`` builds variable-length |
| 18 | + arrays directly. |
| 19 | + |
| 20 | +- **Full indexing for string columns.** ``create_index()`` now works on |
| 21 | + ``utf8()`` and ``dictionary()`` columns via alphabetical ranks — |
| 22 | + ``sort_by`` drops from 424 ms to 7 ms at 1M rows — and scalar comparisons |
| 23 | + are served from the index. |
| 24 | + |
| 25 | +- **New ``blosc2.random`` module**: 42 of NumPy's 43 ``Generator`` methods, |
| 26 | + each chunk generated in parallel with its own seeded ``PCG64`` stream |
| 27 | + (~3x faster than the NumPy path on 100M elements). |
| 28 | + |
| 29 | +- **Slicing up to 1.7x faster.** Plain slice/int keys skip ndindex's general |
| 30 | + machinery (it was 43% of a scattered-read loop); strided steps, ellipsis |
| 31 | + and fancy indexing still use it. |
| 32 | + |
| 33 | +- **String plumbing**: ``from_utf8()``/``to_utf8()`` conversions, |
| 34 | + ``CTable.add_column(values=)``, ``Column.assign()`` on variable-length |
| 35 | + columns, ``StringDType`` dispatch in the array constructors, and DSL |
| 36 | + operands can be native NumPy arrays or pandas ``Series``. |
| 37 | + |
| 38 | +- **Important fixes**: string column indexes returning zero rows at the |
| 39 | + default column width, ``SChunk`` slices for typesizes above 255 bytes |
| 40 | + (upstream, via C-Blosc2 3.3.1), scalar bools in tuple keys now matching |
| 41 | + NumPy, and a batch of miniexpr correctness fixes. |
33 | 42 |
|
34 | 43 | Install it with:: |
35 | 44 |
|
|
0 commit comments