How can the ZSC better empower the Zarr developer community?

[maxrjones]

I was asked to create this issue by the Zarr-Python developers who attended our last sync and encouraged by @joshmoore in zarr-conventions/zarr-conventions-spec#5 (comment). Please understand that these thoughts are my own, quickly and non-comprehensively written down. I state them with extreme appreciation and respect.

Here are some road bumps that I've heard of or experienced myself regarding developers/community members not having the power to take actions:

• developers need to ask ZSC members to add new members to the zarr-developers community organization. this makes it more difficult to manage repository access through teams, which is generally considered best practices. (VirtualiZarr has struggled with this in trying to grow the developer team)
• developers need to ask ZSC members to add/modify/remove calendar events from the Zarr Community Calendar (I am currently struggling with this restriction. I needed to ask the ZSC before scheduling a multiscales call for GeoZarr or organize it through my individual work calendar)
• developers are unable to move or remove notes in Zarr's hackmd team, meaning the workspace has a lot of accidentally created notes and poorly organized notes (I struggled with this restriction a lot at the Zarr summit)
• developers need to ask the ZSC to add billing information (Zarr Python struggled with this when they got funding for GPU based CI)
• developers need to ask the ZSC to add applications to the github organization (Zarr Python struggled with this when they wanted to use a benchmarking service)

I have reacted to these restrictions by creating another github organization for zarr conventions, which is confusing and concerned for adopters (zarr-conventions/zarr-conventions-spec#5, zarr-conventions/zarr-conventions-spec#9). Zarr-Python developers have similarly considered whether it would be better to separate out the implementation from this github organization. I think that would not be good, given that it could be perceived as a rift amongst the community.

Again, thanks for all your work and the invitation to provide constructive feedback @zarr-developers/steering-council.

[d-v-b]

as someone doing a lot of zarr-python maintenance, it's pretty tedious pinging a ZSC member to do basic zarr-python maintenance things. Ideally the two lead zarr-python devs (joe and me) should be able to do our work without requesting stuff from the ZSC.

[rabernat]

I personally see no reason to tie administrative github privileges to the ZSC. I would propose that any active maintainer should get admin rights on the github org. This would unblock most of the issues above.

In Pangeo, we basically gave admin access to anyone who wanted to take on work. This worked well. We never had an issue. When people became inactive, they were removed.

[joshmoore]

Thanks for keeping the conversation going, @maxrjones. This was actually one of the primary things I wanted to discuss at the summit, but I was just there too briefly (sorry about that!).

I think at least partially we're dealing with cultural differences here, which are always a bit trickier since they get to pretty deep seated assumptions. My apologies to the extent that my deep seated assumptions make people feel disempowered.

Briefly some personal thoughts from my side without having spoken to the ZSC on the specific (in my mind largely technical) barriers raised above:

• https://hackmd.io/team/zarr: in my mind, this was mostly a tool of convenience in the early days and was never meant to be proscriptive. I'm in no way a hackmd poweruser and so my thought process before the summit was: "huh, I don't know what any of these permission settings do. I don't know if it will unintentionally expose data that wasn't supposed to be exposed, I'll deal with that later."
• the calendar shown at https://zarr.dev/office-hours/ and elsewhere: This is a bog standard calendar that I think belongs to either me or zarrdevelopers@gmail.com because that's what we set up in the early days. I don't remember a request from anyone to be added with writes, but by all means. Alternatively, by changing the link referenced in https://raw.githubusercontent.com/zarr-developers/zarr-developers.github.io/refs/heads/main/office_hours/index.md etc. Again, there's nothing special about it, it's just what's in place.
• https://github.com/zarr-developers: this is probably the real crux of the matter. All the functions mentioned -- adding members, applications, even billing -- I very much support.

But here's where the cultural difference comes in: where I hesitate, similar to the hackmd issue above, is on the farther reaching implications. And my primary worry is, probably always, about the long-term safety & stability of the spec. And that's not to implicate anyone or express that I don't trust people currently but that I've seen too much happen to make such decisions lightly.

So here's my ask: though I very much support empowerment, I read this issue to mean, "give all zarr developers full rights on the org" (since that's the only way I know how to achieve what you're asking with a single org.) What I'm looking for, in turn, is thoughts from everyone as to the related responsibilities. What are the rules on merging? What changes are in bounds? What is our culture for protecting the spec? We probably all have a feeling for that implicitly, but I think it's worthwhile to make that explicit.

[maxrjones]

I read this issue to mean, "give all zarr developers full rights on the org"

I was intentional in not suggesting solutions, only presenting the problem and its impact. I think @d-v-b's suggestion would likely work for most people - one or two trusted implementers from each primary (i.e., represented on the ZIC) project being owners of the zarr-developers organization.

What are the rules on merging? What changes are in bounds? What is our culture for protecting the spec? We probably all have a feeling for that implicitly, but I think it's worthwhile to make that explicit.

(getting into opinions) I think this should be established on a per-repo basis rather than on the organization as a whole. By adding more owners, you would need to trust those owners to not abuse the expectations on any repository. If you want a middle ground, I prefer a "trust and verify" approach rather than a "don't trust" approach - backup the spec and setup monitoring to make sure the roles are not abused.

[joshmoore]

I was intentional in not suggesting solutions, only presenting the problem and its impact.

Understood and appreciated. 👍

I think this should be established on a per-repo basis rather than on the organization as a whole.

Also agreed, but here's where AFAIK it's not possible to give sufficient repository-level rights to unblock all of the road bumps you've rightly presented.

To the extent that there's interest, after the looming (OME-)Zarr workshop next week, happy to be a part of a task-force/work-group/what-have-you that works on solutions, essentially continuing the spirit of the summit. 🙏🏽

[maxrjones]

Also agreed, but here's where AFAIK it's not possible to give sufficient repository-level rights to unblock all of the road bumps you've rightly presented.

Yes, this is where I think the ZSC would need to trust more individuals to be organization owners and not abuse their powers.

[d-v-b]

But here's where the cultural difference comes in: where I hesitate, similar to the hackmd issue above, is on the farther reaching implications. And my primary worry is, probably always, about the long-term safety & stability of the spec. And that's not to implicate anyone or express that I don't trust people currently but that I've seen too much happen to make such decisions lightly.

What is the threat model here? That someone, e.g. max, is given admin rights, and then he opens PRs against the spec, then someone else approves it (presumably the zarr-specs PRs don't allow the PR author to merge), and then the spec is broken, and for some reason it can't be fixed? Has this ever happened in the history of this project, and if so was the solution to only have a very small number of admins? I would appreciate concrete examples because I am not aware of anything like this happening.

What I can say is that over in zarr-python we have a real risk -- we might add two malicious actors to the zarr-python core dev team, and this pair might open PRs that do bad stuff like delete user data or upload their s3 credentials to a private server. This is a real risk that I think about, because unlike someone changing the spec, zarr-python is real code that people run on their computers. But this very real concern doesn't stop us from adding core devs as we see fit, because we have a review process and we trust each other. Maybe this example is useful for the zarr steering council?

[joshmoore]

For what it's worth, @d-v-b, I worry about that as well. :) But I agree with you, this isn't about malicious text in the spec. The threat model that I'm concerned here is organizational. Perhaps I'm alone in that, but as I said, it's my cultural background.

We clearly have discussions that need to be had, but I'm not sure we have to answer everything immediately. Personally, I'd like to cool down this thread and perhaps give some others space for commenting.

[maxrjones]

The threat model that I'm concerned here is organizational.

would you be willing to explain what this means to me?

[joshmoore]

would you be willing to explain what this means to me?

Fair. For the sake of clarity, one more comment from my side tonight :) --

The worst case scenario, the thing that it's even fraught to mention, is some kind of rift leading to a fork. But there's an entire spectrum, including for example the creation of competing repos (zarr-conventions vs zarr-extensions vs zarr-attributes vs zarr-plugins vs ...). Of course, there's the reverse issue of multiple orgs which you brought up to begin with, @maxrjones, and so I'm not saying minimal org owners is perfect or a must.

I'm also not saying any of that has happened before nor that it is likely to happen, but I'm asking for help with the mental load of defining our frameworks, policies, and even just values on how things here function. For me that's the responsibility that comes with the rights, but I also think having that shared representation is part of what makes a community.

[maxrjones]

I don't really understand how github org ownership relates to forking and am less impacted by the current model than some of the power-maintainers here. For those reasons, I'm going to leave it to others to iterate on the model of github ownership/billing/security management for Zarr.

The two components that frequently impact my ability to contribute are the hackmd and google calendar restrictions. Would you please add me as an admin on the hackmd team and give me permissions to change the Zarr public calendar @joshmoore?

[rabernat]

I'd say the greatest risk to Zarr today is that smart developers get so frustrated by our self-imposed bureaucracy and the limits it imposes on the development process that they lose motivation and move on to other things.

[normanrz]

Hi everybody, I agree that we should try to reduce bottlenecks in our processes. While I have no objection to adding a few more people as admins to the org, I don't think adding everyone will solve the issues.

I think what is missing is a "secretary general" role in Zarr, who manages and reviews permissions in our systems and can help with ad-hoc issues. This role could be held by someone from the ZSC or someone else from the community. It could also be a rotating role. The goal would be to have a single person who feels responsible and empowered for these tasks and is responsive to requests from the community.
Right now, the ZSC is comprised of 5 people. I don't think it is always clear who is responsible to take actions on requests and, therefore, they get stalled.

[dstansby]

it's not possible to give sufficient repository-level rights to unblock all of the road bumps you've rightly presented.

I think it is possible to give repository-level rights. In particular, with respect to:

developers need to ask ZSC members to add new members to the zarr-developers community organization.

one can create a team (these probably already exist in zarr-developers org) and then set per-repository permission for this team. Members of a team can then be designated "Maintainer" for a team, and gain rights to add/remove members from the team, with no additional rights to any other organisation repositories.

I would also like to echo #50 (comment), and I hope those who have new time, energy, and enthusiasm to help make Zarr prosper are empowered and brought along for the ride instead of adding extra layers of beuracracy.