Support fsspec

[FrancescAlted]

This gives access to the remote protocols that fsspec provides.

[Copilot]

Pull request overview

Adds fsspec-backed remote container access, including whole-object transfers, local caching, and lazy range reads.

Changes:

• Adds remote open/save support and cache invalidation.
• Introduces concurrent lazy chunk fetching through FsspecNDSource.
• Adds packaging, tests, examples, and documentation.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/blosc2/core.py	Adds fsspec detection, caching, and tensor I/O.
src/blosc2/schunk.py	Dispatches remote opens and lazy proxies.
src/blosc2/proxy.py	Adds persistent caches and FsspecNDSource.
src/blosc2/ndarray.py	Supports remote NDArray.save().
src/blosc2/storage.py	Rejects incrementally backed remote containers.
src/blosc2/__init__.py	Exports FsspecNDSource.
tests/test_fsspec.py	Tests remote I/O, caching, and lazy reads.
tests/ndarray/test_proxy.py	Tests persistent proxy cache reuse.
pyproject.toml	Adds fsspec dependencies.
RELEASE_NOTES.md	Documents the feature.
plans/fsspec-support.md	Records design and implementation details.
examples/ndarray/rw-fsspec.py	Demonstrates remote I/O modes.
examples/ndarray/concurrent-fsspec.py	Demonstrates concurrent fetching.
doc/reference/fsspecndsource.rst	Adds API reference documentation.
doc/reference/classes.rst	Adds the new reference page.
doc/getting_started/installation.rst	Documents installation and usage.

Suppressed comments (2)

src/blosc2/core.py:685

• This manifest can serve stale directory data after a same-size overwrite. Backends such as memory:// expose neither mtime nor LastModified, so an unchanged filename and size produce the same manifest even when its bytes changed. Use each filesystem's ukey() (as the lazy cache already does) so content/version identity participates in invalidation.

    listing = json.dumps(
        {
            name: (entry.get("size"), entry.get("mtime") or entry.get("LastModified"))
            for name, entry in sorted(fs.find(path, detail=True).items())
        },

src/blosc2/proxy.py:810

• The actual async-driver branch is untested: test_lazy_afetch uses MemoryFileSystem, whose async_impl is false, and therefore only exercises the blocking fallback above. Add a fake async fsspec filesystem test that reaches _cat_file and verifies range arguments, overlapping requests, and propagation of fetch errors before relying on this path for S3/GCS.

        data = await self._fs._cat_file(self._path, start=offset, end=end)
        return data[: struct.unpack("<i", data[12:16])[0]]

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

src/blosc2/core.py:761

• An explicit mode="r" is silently discarded here, after which the remote object is overwritten. Local saves reject writes in reading mode, so the fsspec path must validate the mode before removing it rather than bypassing that protection.

    remote_urlpath = kwargs.get("urlpath") if is_fsspec_url(kwargs.get("urlpath")) else None
    if remote_urlpath is not None:
        del kwargs["urlpath"]
        kwargs.pop("mode", None)

src/blosc2/proxy.py:779

• This opens a buffered fsspec file to read only the frame index. On s3fs the default block size is 50 MiB, so the initial 24-byte read can fetch a 50 MiB block and the seek to the index can fetch another, defeating the advertised small lazy-open transfer. Parse the header/index with exact cat_file(start=..., end=...) range reads, or explicitly disable buffering.

        with fs.open(path, "rb") as f:
            raw, header, self._offsets = _read_frame_index(f)

src/blosc2/ndarray.py:7027

• Normalization happens before a mapping passed via storage= is expanded. Consequently blosc2.zeros(..., storage={"urlpath": "memory://..."}) bypasses both Storage.__post_init__ and this normalization, reaching the C layer instead of producing the new explicit fsspec error; file:// paths in such mappings also remain unnormalized. Normalize/validate the merged storage URL (or construct a Storage from mappings) before returning kwargs.

    if kwargs.get("urlpath") is not None:
        # A Storage instance normalizes its own; a bare kwarg has to be done here,
        # since it takes precedence over the defaults built from it below
        kwargs["urlpath"] = normalize_urlpath(kwargs["urlpath"])

src/blosc2/core.py:631

• A non-local file authority is concatenated without the leading //. On Windows, file://server/share/a.b2nd therefore becomes the relative path server\\share\\a.b2nd instead of the UNC path \\\\server\\share\\a.b2nd (and is relative on POSIX too). Preserve // for host authorities while retaining the special C: drive handling.

This issue also appears on line 758 of the same file.

        # A Windows drive lands in netloc for the two-slash form, `file://C:/x`
        prefix = parsed.netloc if parsed.netloc.lower() not in ("", "localhost") else ""
        return urllib.request.url2pathname(prefix + parsed.path)

plans/fsspec-support.md:31

• These links use a developer's absolute /Users/faltet/... paths, so they resolve to broken links when this plan is viewed from the repository. The same pattern recurs throughout the document; replace all of them with repository-relative links.

Today there is no S3 support at all. `s3fs` appears in the repo only in
[bench/ndarray/download_data.py](/Users/faltet/blosc/python-blosc2/bench/ndarray/download_data.py)
and in the `dev` dependency group of
[pyproject.toml](/Users/faltet/blosc/python-blosc2/pyproject.toml). Passing
`s3://...` to `blosc2.open()` falls through the store-probing branches in
[src/blosc2/schunk.py](/Users/faltet/blosc/python-blosc2/src/blosc2/schunk.py)

[FrancescAlted]

Good catch, and it was a real gap — fixed in bf87015.

blosc2.open(url, lazy=True, cache_storage=...) did compare identity: _lazy_fsspec_proxy() removed the cache when _cache_stamp(path) != src.stamp (fs.ukey(), which is fsspec's own token rather than metadata we guess a backend exposes). But the hand-built form the FsspecNDSource docstring recommends —

src = blosc2.FsspecNDSource("s3://bucket/big.b2nd")
a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a")

— went straight to Proxy and skipped that check entirely, which is exactly the hole you describe.

So the check moved into Proxy itself: __init__ stamps the cache with src.stamp for any source that has one, and _reopen_cache() refuses a cache whose stamp disagrees. _lazy_fsspec_proxy() no longer stamps by hand; its remove_urlpath() is what turns that refusal into a refetch. Covered by test_handbuilt_proxy_rejects_a_stale_cache.

On the second half of the suggestion — rejecting adoption when the source cannot provide an identity — I deliberately did not do that. Geometry-based adoption for plain ProxySource/ProxyNDSource implementations is the documented behaviour of mode="a" (test_reuse_cache_across_runs), not an oversight; a user source has no identity to offer, and refusing it would remove the feature rather than harden it. Sources that can name themselves are now checked, and those are the ones where a replaced source is actually reachable.

[FrancescAlted]

Correct — fixed in 5aec265.

NDArray.save() now runs the same guard local persistence does before touching the remote object:

# An object store takes the whole thing at once, and always replaces,
# but reading mode still forbids a write
blosc2_ext.check_access_mode(urlpath, kwargs.pop("mode", "w"))

The identical bug in pack_tensor() (which you flagged as a suppressed comment on core.py:761) is fixed the same way. Both are covered by test_save_to_url_rejects_reading_mode.

[FrancescAlted]

Thanks — the two inline comments are answered in their threads. The five suppressed ones were all valid too, and all five are fixed; since they have no thread to reply to, here they are one by one.

core.py:761 — mode="r" discarded in pack_tensor(). Same bug as the ndarray.py one, fixed the same way: blosc2_ext.check_access_mode(remote_urlpath, kwargs.pop("mode", "a")). Test: test_save_to_url_rejects_reading_mode.

proxy.py:779 — buffered handle for the frame index. Right about the cost: on s3fs the 24-byte prefix read pulls a 50 MiB block and the seek to the index pulls another, which is precisely what lazy=True exists to avoid. The index now goes through exact ranges — a small _RangeReader shim giving _read_frame_index() the seek/read pair it wants, backed by fs.cat_file(path, start=, end=), the same call get_chunk() already used. test_lazy_open_never_opens_a_handle monkeypatches _open to fail, so a regression to a buffered handle breaks the build rather than just getting slower.

ndarray.py:7027 — normalization before the storage= mapping is expanded. Confirmed, and the consequence was worse than cosmetic: blosc2.zeros(..., storage={"urlpath": "memory://q.b2nd"}) died with an opaque RuntimeError: Could not build zeros array instead of the explicit fsspec ValueError, and file:// in a mapping stayed unnormalized. The normalization moved to after the merge and now borrows Storage.__post_init__ rather than duplicating it:

if kwargs.get("urlpath") is not None:
    # A Storage instance normalized (and vetted) its own on construction; a bare
    # kwarg or a `storage=` mapping never reached __post_init__, so borrow it here
    kwargs["urlpath"] = blosc2.Storage(urlpath=kwargs["urlpath"]).urlpath

Test: test_storage_mapping_is_normalized.

core.py:631 — non-local authority concatenated without //. Fixed, with one correction to the suggested shape. Prepending // unconditionally is right on Windows but wrong on POSIX: Python 3.14's urllib.request.url2pathname raises URLError: file:// scheme is supported only on localhost for a non-local authority, because a UNC host has no POSIX equivalent to normalize to. So the drive case keeps its bare C:, a host gets // on Windows, and on POSIX it raises a ValueError naming the host instead of silently producing a relative path. test_normalize_file_url_with_a_host asserts both branches.

plans/fsspec-support.md:31 — absolute /Users/faltet/... links. All 19 stripped to repository-relative.

---

The review also surfaced nothing about the Windows CI failure, which was mine: test_normalize_file_url compared separators rather than layout, so \tmp\a.b2nd failed the "/tmp/a.b2nd" in ... assertion. It now compares PurePath(...).as_posix().

A separate review pass found seven more, fixed in the same two commits: a cached .b2e store opening as a bare SChunk (fsspec's hashed cache name dropped the extension blosc2.open() dispatches on); lazy=True on a zero-length array reading the trailer as an offsets chunk; a half-written cache being fatal to every later open rather than discarded; max_concurrency= and Proxy constructor kwargs being silently ignored in cases where they cannot apply; the cache-mismatch message naming (shape, dtype, chunks, blocks) while printing three byte counts on the SChunk branch; and an O(n) numpy scan per chunk in fetch().

All 35 checks are green, including Windows. Locally: 9399 passed / 29 skipped, plus the 8 moto-backed S3 tests in tests/test_fsspec_s3.py.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/blosc2/proxy.py:358

• The fetched bitmap now overrides the cache's actual special state. After a chunk is fetched, calling proxy.schunk.update_special(n, SpecialValue.UNINIT) leaves this bit set, so the next access serves the uninitialized cache chunk instead of refetching it. That regresses the documented cache-eviction behavior of SChunk.update_special (src/blosc2/schunk.py:757-760). Keep explicit eviction synchronized with this bitmap and cover the fetch → evict → refetch path.

    def _missing_chunks(self, item) -> list[int]:
        """The chunks *item* touches, minus those already in the cache."""
        nchunks = self._schunk_cache.nchunks
        # Full realization when item is (), else only the chunks it intersects
        wanted = range(nchunks) if item == () else list(blosc2.get_slice_nchunks(self._cache, item))
        return [int(n) for n in wanted if not self._fetched[n // 8] >> (n % 8) & 1]

src/blosc2/proxy.py:242

• This paragraph states that a same-geometry changed source is adopted immediately after citing FsspecNDSource.stamp, but stamped sources are actually rejected by _reopen_cache. Clarify that geometry-only adoption applies only to sources without an identity; a stamp mismatch raises.

            Geometry is all that is checked unless the source can name the exact
            bytes it reads, as :ref:`FsspecNDSource` does with its ``stamp``: a
            source whose contents changed underneath while its geometry did not
            is adopted, and the cache keeps serving what the earlier run fetched.
            Pass ``mode="w"`` when the source may have been rewritten.