Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support decompressing Chunks with Blosc AUTOSHUFFLE (-1) #8381

Open
fm3 opened this issue Feb 5, 2025 · 0 comments
Open

Support decompressing Chunks with Blosc AUTOSHUFFLE (-1) #8381

fm3 opened this issue Feb 5, 2025 · 0 comments

Comments

@fm3
Copy link
Member

fm3 commented Feb 5, 2025

When decompressing zarr chunks we got this message from JBlosc:

blosc: shuffle type not supported: '-1'; expected one of 0 (NOSHUFFLE),1 (BYTESHUFFLE),2 (BITSHUFFLE)

According to this document, -1 is AUTOSHUFFLE. There, bit-shuffle will be used for buffers with itemsize 1, and byte-shuffle will be used otherwise. The default is SHUFFLE.

If AUTOSHUFFLE, bit-shuffle will be used for buffers with itemsize 1, and byte-shuffle will be used otherwise. The default is SHUFFLE.

We should either update to a Blosc lib that supports this AUTOSHUFFLE directly or mutate its input so that we do the byteshuffle-bitshuffle switch as described by this specification.

Context and example zarr dataset: https://scm.slack.com/archives/C02H5T8Q08P/p1738766454436339

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant