Skip to content

Commit d434ad2

Browse files
FrancescAltedclaude
andcommitted
Record what route A became
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 4e1db6c commit d434ad2

1 file changed

Lines changed: 30 additions & 2 deletions

File tree

plans/fsspec-blocks.md

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,37 @@
11
# Block-Granular Downloads For `blosc2.open(url, lazy=True)`
22

3-
Analysis only — nothing implemented. Written 2026-08-16 on branch
4-
`fsspec-support-plan`, after
3+
Written 2026-08-16 on branch `fsspec-support-plan`, after
54
[plans/fsspec-support.md](fsspec-support.md) phase 3 shipped.
65

6+
**Status: route A is implemented** (2026-08-17, branch `fsspec-blocks`), with
7+
A.3a as recommended. What the code does differently from the design below:
8+
9+
- **The whole-chunk fallback needs no header read at all.** The plan put the
10+
wanted-bytes test after the layout read; measuring made it clear that the
11+
case worth avoiding — a slice wanting every block of a chunk — is decided by
12+
geometry alone. `FsspecNDSource.wants_blocks()` answers from the block count
13+
and from the chunk extent the frame's offsets already give, so a chunk that
14+
does not want blocks costs exactly one request, as before. That is what makes
15+
block fetching the default rather than a flag.
16+
- **Mitigation 2 (persist the layouts) is not in.** The layouts are memoized on
17+
the source, which covers repeated slicing within a session; keeping them in
18+
the cache's vlmeta across runs is a `ponytail:` note in the code, to do when
19+
someone reopens the same array often enough to care.
20+
- **`afetch()` stays chunk-granular.** Driving two dependent waves through
21+
`asyncio.gather` buys nothing that the sync path does not already have, and
22+
the sync path is what `__getitem__` uses. Documented rather than hidden.
23+
- **A cache with no usable bitmap is refetched, not guessed at.** The chunk path
24+
reconstructs "what is already here" by scanning for non-special chunks; a
25+
spliced chunk is not special, so in block mode that guess would serve zeros
26+
for blocks that never arrived. Found by a test that reopened a cache and read
27+
a new slice out of an already-touched chunk.
28+
- Measured end to end afterwards with `--moto`: on 13 MB chunks of 134 blocks, a
29+
point read moves 0.10 MB instead of 13.15 MB — 3.1x faster over an in-region
30+
network, 7.0x over a transatlantic one, 1.0x when the slice wants every block.
31+
32+
Route B (the io-callback bridge) stays unbuilt, and stays the answer for the
33+
formats route A cannot reach.
34+
735
## The question
836

937
`blosc2.open(url, lazy=True)` returns a `Proxy` over `FsspecNDSource`

0 commit comments

Comments
 (0)