Re-implement netCDF3 parser without Kerchunk - #1086
Open
TomNicholas wants to merge 7 commits into
Open
Conversation
TomNicholas
added
the
references generation
TomNicholas
temporarily deployed
to
test-release
August 13, 2026 19:50 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
TomNicholas
commented
Aug 13, 2026
TomNicholas
temporarily deployed
to
test-release
August 13, 2026 19:52 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
NetCDF3Parser went through kerchunk.netCDF3.NetCDF3ToZarr, which subclasses scipy.io._netcdf.netcdf_file and overrides a private method. Only the file header is ever needed, and the netCDF classic header is a small, fully specified big-endian grammar that ends every variable's record with that variable's byte offset -- exactly what a ChunkManifest needs. Parse it directly instead. This drops kerchunk and scipy as dependencies (the virtualizarr[netcdf3] extra is now empty, and no longer pulls in virtualizarr[remote]), reads the header through the ObjectStoreRegistry like every other native parser rather than opening the file with fsspec, and adds support for CDF-5, which scipy cannot read at all. Record variables are interleaved on disk, so they produce one chunk reference per record per variable. These are now built as vectorized numpy arrays straight into a ChunkManifest, rather than being materialized as an intermediate kerchunk references dict and parsed back out. On a 6.4 MB file with 20 record variables of 4000 records each, open_virtual_dataset goes from ~997 ms to ~63 ms, with all 80,000 chunk references identical to the ones the kerchunk path produced. Also fixes zarr-developers#982: kerchunk excludes _FillValue from the attributes it copies, so sentinel values survived into the loaded data instead of decoding to NaN. Tests cover all three CDF versions against what xarray reads natively, plus the corners the format hides: CDF-5's 8-byte dimension ids, the unpadded record stride a file with exactly one record variable uses, streaming numrecs recovered from file size, char and scalar variables, and malformed headers.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1086 +/- ##
==========================================
+ Coverage 89.67% 89.96% +0.29%
==========================================
Files 41 41
Lines 2682 2850 +168
==========================================
+ Hits 2405 2564 +159
- Misses 277 286 +9
🚀 New features to boost your workflow:
|
TomNicholas
marked this pull request as ready for review
August 13, 2026 19:55
TomNicholas
temporarily deployed
to
test-release
August 13, 2026 19:55 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
|
@TomNicholas This is amazing. An hour after you made this PR I was about to start figuring out how to deal with CDF5 files in kerchunk, which this appears to fix (I just tried it out). Thank you so much!!! |
…teral The pixi hdf5-lib feature already provides conda-forge netcdf4 to every test environment, so adding netCDF4 to the dev dependency-group was redundant. On the free-threaded py314 environments (min-deps, test, test-py314) the pip wheel's bundled HDF5 collided with the conda one, breaking hdf5plugin's blosc filter and erroring out the blosc_lz4 HDF filter tests. py312 and py313 resolved a compatible wheel, which is why only the py314 jobs failed. Also annotate the netCDF3 format strings as a Literal, so the format argument matches what the netCDF4 stubs expect and mypy passes.
TomNicholas
temporarily deployed
to
test-release
August 13, 2026 23:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
The netcdf3 extra is now empty, since the parser needs nothing beyond the core dependencies, so remove it outright rather than leave an empty placeholder. This is a breaking change: 'pip install virtualizarr[netcdf3]' no longer resolves. Also drop it from all_parsers and from every pixi environment. netCDF4 is needed only to *write* the netCDF3 test files -- it is the only writer that can emit CDF-5, which neither xarray nor scipy supports. Rather than import it unconditionally in conftest, put it behind a netcdf4_lib fixture that skips when it is absent, so the tests that build their files by hand still run without it. It stays a conda dependency of the hdf5-lib feature, since the pip wheel bundles a libhdf5 that collides with the conda one on py314.
TomNicholas
temporarily deployed
to
test-release
August 14, 2026 00:03 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
TomNicholas
changed the title
… docs The netCDF3 parser no longer wraps kerchunk, so FITS and HDF4 are the only parsers left using that approach internally. Group netCDF3 with HDF5 as a parser that started out kerchunk-based and has since been reimplemented natively.
TomNicholas
temporarily deployed
to
test-release
August 17, 2026 12:33 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the current implementation of the
NetCDF3Parserwith a new standalone implementation which does not require kerchunk or scipy. This one is faster, can understand a wider range of netCDF3 format variants, fixes at least one bug (#982), and has fewer dependencies.🤖
NetCDF3Parser went through
kerchunk.netCDF3.NetCDF3ToZarr, which subclassesscipy.io._netcdf.netcdf_fileand overrides a private method. Only the file header is ever needed, and the netCDF classic header is a small, fully specified big-endian grammar that ends every variable's record with that variable's byte offset — exactly what a ChunkManifest needs.Parsing it directly drops kerchunk and scipy as dependencies (virtualizarr[netcdf3] is now empty and no longer pulls in virtualizarr[remote]), reads the header through the ObjectStoreRegistry like every other native parser, and adds CDF-5 support, which scipy cannot read at all.
Record variables are interleaved on disk, so they produce one chunk reference per record per variable. These are now built as vectorized numpy arrays straight into a ChunkManifest instead of going through an intermediate kerchunk refs dict. On a 6.4 MB file with 20 record variables × 4000 records, open_virtual_dataset goes from ~997 ms to ~63 ms (15.8x), with all 80,000 chunk references identical to the kerchunk path.
Closes #982 — kerchunk excludes _FillValue from the attributes it copies, so sentinels survived into loaded data instead of decoding to NaN.
Tests cover all three CDF versions against what xarray reads natively, plus the corners the format hides: CDF-5's 8-byte dimension ids, the unpadded record stride when a file has exactly one record variable, streaming numrecs recovered from file size, char and scalar variables, and malformed headers.