|
1 | 1 | # Block-Granular Downloads For `blosc2.open(url, lazy=True)` |
2 | 2 |
|
3 | | -Analysis only — nothing implemented. Written 2026-08-16 on branch |
4 | | -`fsspec-support-plan`, after |
| 3 | +Written 2026-08-16 on branch `fsspec-support-plan`, after |
5 | 4 | [plans/fsspec-support.md](fsspec-support.md) phase 3 shipped. |
6 | 5 |
|
| 6 | +**Status: route A is implemented** (2026-08-17, branch `fsspec-blocks`), with |
| 7 | +A.3a as recommended. What the code does differently from the design below: |
| 8 | + |
| 9 | +- **The whole-chunk fallback needs no header read at all.** The plan put the |
| 10 | + wanted-bytes test after the layout read; measuring made it clear that the |
| 11 | + case worth avoiding — a slice wanting every block of a chunk — is decided by |
| 12 | + geometry alone. `FsspecNDSource.wants_blocks()` answers from the block count |
| 13 | + and from the chunk extent the frame's offsets already give, so a chunk that |
| 14 | + does not want blocks costs exactly one request, as before. That is what makes |
| 15 | + block fetching the default rather than a flag. |
| 16 | +- **Mitigation 2 (persist the layouts) is not in.** The layouts are memoized on |
| 17 | + the source, which covers repeated slicing within a session; keeping them in |
| 18 | + the cache's vlmeta across runs is a `ponytail:` note in the code, to do when |
| 19 | + someone reopens the same array often enough to care. |
| 20 | +- **`afetch()` stays chunk-granular.** Driving two dependent waves through |
| 21 | + `asyncio.gather` buys nothing that the sync path does not already have, and |
| 22 | + the sync path is what `__getitem__` uses. Documented rather than hidden. |
| 23 | +- **A cache with no usable bitmap is refetched, not guessed at.** The chunk path |
| 24 | + reconstructs "what is already here" by scanning for non-special chunks; a |
| 25 | + spliced chunk is not special, so in block mode that guess would serve zeros |
| 26 | + for blocks that never arrived. Found by a test that reopened a cache and read |
| 27 | + a new slice out of an already-touched chunk. |
| 28 | +- Measured end to end afterwards with `--moto`: on 13 MB chunks of 134 blocks, a |
| 29 | + point read moves 0.10 MB instead of 13.15 MB — 3.1x faster over an in-region |
| 30 | + network, 7.0x over a transatlantic one, 1.0x when the slice wants every block. |
| 31 | + |
| 32 | +Route B (the io-callback bridge) stays unbuilt, and stays the answer for the |
| 33 | +formats route A cannot reach. |
| 34 | + |
7 | 35 | ## The question |
8 | 36 |
|
9 | 37 | `blosc2.open(url, lazy=True)` returns a `Proxy` over `FsspecNDSource` |
|
0 commit comments