Make create_array signatures consistent

[d-v-b]

This PR ensures that the various invocations of create_array are consistent. For reference, we have 4 ways to call create_array:

• zarr.core.array.create_array (the actual function that does stuff)
• zarr.api.synchronous.create_array (synchronous wrapper around the async function)
• zarr.core.group.AsyncGroup.create_array (method on AsyncGroup class that invokes create_array)
• zarr.core.group.Group.create_array (synchronous wrapper around the AsyncGroup method)

All of these functions should have consistent signatures, but in main they don't, and a big part of this is missing tests. So this PR adds some tests that check that certain pairs of functions have identical parameters (we can't force the return types to match, because the async functions will return coroutines). To make the tests pass, this PR also ensures that all of the invocations of create_array have parameters that are consistent.

I say "consistent" because, when invoking Group.create_array, that method does not take a store argument, because we have one already from the group instance. Also, Group.create_array was recently given an extra keyword argument (compressor) that we need to deprecate. So the test that compares zarr.core.array.create_array with zarr.core.group.AsyncGroup.create_array only checks that all of create_array parameters are present in the group method.

closes #2810

[d-v-b]

worth noting that this changes some defaults (we were using a default fill_value of None in some places, and 0 in other places, for example). When in doubt, I picked the default that seemed more defaulty.

[LDeakin]

worth noting that this changes some defaults (we were using a default fill_value of None in some places, and 0 in other places, for example). When in doubt, I picked the default that seemed more default.

fill_value = None seems a more sensible default to me, with some docs somewhere explaining how this maps to a default fill value for each data type. 0 needs conversion for any non-integer data type, and this is exactly what caused #2792 (comment) in zarr-python 2.

[d-v-b]

worth noting that this changes some defaults (we were using a default fill_value of None in some places, and 0 in other places, for example). When in doubt, I picked the default that seemed more default.

fill_value = None seems a more sensible default to me, with some docs somewhere explaining how this maps to a default fill value for each data type. 0 needs conversion for any non-integer data type, and this is exactly what caused #2792 (comment) in zarr-python 2.

That works for me. In main I think we have a mix of 0 and None floating around, I think I chose 0 for this PR because it seemed like the majority, but I'm happy to switch to None here

[d-v-b]

with 642272d the default fill value is None for the array creation funcs.

[dstansby]

👍 for this in general - I left one request for a more verbose changelog entry, and one question.

[dstansby]

Thanks for the changelog update, looks great to me. I left one more suggestion - I did a bit of testing and this doesn't seem to change the fill value that's set on the arrays (which is good, but I think worth reassuring users)

[d-v-b]

Thanks for the changelog update, looks great to me. I left one more suggestion - I did a bit of testing and this doesn't seem to change the fill value that's set on the arrays (which is good, but I think worth reassuring users)

how did you check this? because for zarr v2 data, it should make a difference -- None is a valid v2 fill value, and it's not the same as 0

[dstansby]

I didn't check for v2 🙈 . If it does make a difference, that should be made very clear in the changelog (and we should think about putting this PR in a non-bugfix release since it's a breaking change?)

[LDeakin]

how did you check this? because for zarr v2 data, it should make a difference -- None is a valid v2 fill value, and it's not the same as 0

I didn't check for v2 🙈 . If it does make a difference, that should be made very clear in the changelog (and we should think about putting this PR in a non-bugfix release since it's a breaking change?)

I'd hope that since the below PRs, the metadata of the Zarr V2 data with fill_values=None will have a null fill value but actual stored fill values will match the V3 default rather than being completely undefined.

• Use implicit fill values for zarr v2 #2274
• fix: implicit fill value initialisation #2799

[d-v-b]

@LDeakin that was my impression as well. I think for v2 data the contents of the array metadata will be sensitive to the None / 0 distinction, but at runtime the actual array data should be the same in zarr-python. but this might vary in other implementations.

[LDeakin]

I think for v2 data the contents of the array metadata will be sensitive to the None / 0 distinction, but at runtime the actual array data should be the same in zarr-python. but this might vary in other implementations

Indeed! To be honest, I think null fill values in Zarr V2 were a bit of a misstep anyway, mainly because undefined values in partially written chunks cannot be distinguished from real data. This could be a good opportunity for zarr-python to stop writing null fill values altogether in Zarr V2 metadata and just write the default fill value. What do you think?

[d-v-b]

The nullable fill value is also clunky because zarr v2 supports the python Object dtype, and None is a valid instance of the Object dtype, which makes it impossible to distinguish "no fill value" from "the fill value is the literal None" for arrays with that dtype (or any other dtype that admits None as a value).

We would have to get a measure of the impact on users, but if it doesn't inconvenience a lot of people then I would definitely support dropping the creation of arrays with a null fill value, at least for numeric types

[dstansby]

So to clarify, this PR is a breaking one for writing v2 data, which changes what is written as the fill value if the user doesn't specify it? And this changed test is the manifestation of that breaking change? Is this something we want to avoid breaking?

(sorry for all the questions, just trying to understand 😄 )

[d-v-b]

it's a change to how v2 metadata is written. the actual array data will be written as before. Because the signatures of the affected functions are inconsistent, there is no way to achieve the goal of this PR without changing the default behavior of some of those functions, which will result in different metadata being written to disk as an effect.

[d-v-b]

so to your questions: it's breaking if someone relied on the default metadata document produced by some, but not all, of our array creation routines. And no, we do not want to avoid breaking, because the cost of a confusing, inconsistent API is worth paying in this case

[dstansby]

👍 to all that. In which case this is a backwards incompatible API change, and we promise to increment the major version number for backwards incompatible API changes. So, what do we want to do - bump the version number v4, or re-write the versioning policy?

I'm guessing it's the latter... which is a pain, but I think we should be honest and transparent to users if we're going to put backwards incompatible API changes in minor releases.

[d-v-b]

I think we should adjust the versioning policy, and also release the changes in this PR in a minor (non-patch) release. We were already planning on making breaking changes without incrementing the major version number -- deprecated functions like create were slated to be removed long before a 4.x release. So I think the versioning policy needs updating.

[dstansby]

Sorry to be a bit of a pain, but I'm going to put a request changes on this until we've resolved how we're going to bump the version number (and/or update our versioning policy) for this change. See #2819 (comment) for more context.

[d-v-b]

that works for me! would you like to open an issue about the versioning policy, or should I?

[dstansby]

I can 👍