Raise when to_zarr(append_dim=...) would mislabel data - #11505
Open
ghostiee-11 wants to merge 3 commits into
Open
Raise when to_zarr(append_dim=...) would mislabel data#11505ghostiee-11 wants to merge 3 commits into
ghostiee-11 wants to merge 3 commits into
Conversation
to_zarr writes the appended block in the order given by the dataset being written and never aligns it to the store. When a coordinate that does not have append_dim among its dimensions disagrees with the store, data is silently mislabelled: mode="a" overwrites the coordinate and relabels the data already written, while mode="a-" keeps the store's coordinate and mislabels the block being appended. Compare such coordinates against the store before writing anything and raise if they differ. Values are compared as encoded, so a lossy or value-changing encoding does not raise spuriously. Data variables are unaffected and keep the documented mode="a"/mode="a-" behaviour. Fixes pydata#11101
github-actions
Bot
added
topic-backends
topic-zarr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
to_zarr(append_dim=...)writes the new block in the incoming order without aligning it to the store, so a mismatched coordinate silently mislabels data:mode="a"relabels what's already there,mode="a-"mislabels what's appended. Now raises before writing.Checklist
whats-new.rstAI Disclosure