Repo for an AI experiment

[ameukam]

Describe the issue

We will like to get a new GitHub repository to host the codebase for an experimental tool that aims to improve the maintainer experience using Deep Learning techniques. This tool will leverage different AI tools to automate and enhance various aspects of repository maintenance, such as issue triaging, pull request review, and possibly documentation generation.

/assign @justinsb @ameukam

[ameukam]

/sig contribex

[k8s-ci-robot]

@ameukam: The label(s) sig/contribex cannot be applied, because the repository doesn't have them.

In response to this:

/sig contribex

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ameukam]

/sig contributor-experience

[ameukam]

cc @kubernetes/sig-contributor-experience-leads
We reached out the different leads for feedback. Hopefully we will get consensus to open file a request for repo creation.

[palnabarun]

Thinking out aloud, how about using kubernetes-sigs/maintainers for this? You wouldn't need a separate repo then. The said repo has some existing tooling to help maintainers maintain the OWNERS files.

[dims]

works @palnabarun ! (or hydrophone repo is fine as well )

[ameukam]

Thinking out aloud, how about using kubernetes-sigs/maintainers for this? You wouldn't need a separate repo then. The said repo has some existing tooling to help maintainers maintain the OWNERS files.

We're mostly looking for autonomy in terms of PR approvals and CI setup. We don't have a proper definition of success for this experiment so I don't know how complex the code base will look like in the future. An existing repo is fine but I don't want to confuse any new contributor.

[palnabarun]

kubernetes-sigs/maintainers seems to be a nice option then. Both tools (maintainers and the one you are proposing) have similar reasons to exist.

[ameukam]

SGTM. I guess we need to wait for the other TLs to chime in ?

[Priyankasaggu11929]

Hello folks (sorry I was out last week, just catching up & continuing the discussions from Jan, 2025 ContribEx meetings.)

Not blocking on the request for using k-sigs/maintainers repo, but I'm still trying to understand the scope of the overall request, since we have 2 parallel ongoing conversations (which seems similar or converging, but @ameukam you mentioned they're not - so want to check the status at this point):

• request for new repo for AI experiments, by @ameukam
• new maintainers-tool working group proposal, by @justinsb
  • @justinsb, checking if the discussions within this thread, around using kubernetes-sigs/maintainers repo, also aligns with your proposal for new maintainers-tool working group?
  • If we're agreeing to use kubernetes-sigs/maintainers (I'm assuming yes, considering PR k-sigs/maintainers#26), do you still plan to create the new working group? Or that's no longer needed?

---

Also some umbrella questions regarding the scope of AI experiments within the repo:

We don't have a proper definition of success for this experiment so I don't know how complex the code base will look like in the future.

Can we define the "out of scope" items for the moment?
(It's ok to keep revising the list, but to give some boundaries to the experimentation for all current & future folks to be involved?)

---

(further, following up from the contribex meetings notes)
@mfahlandt – did you have any other discussions with k8s-infra group about this?
@ameukam – I remember you mentioning this is a standalone project by you (with not your k8s-infra chair hat on), so checking if there is any third conversation happening as well within k8s-infra group?

[ameukam]

I remember you mentioning this is a standalone project by you (with not your k8s-infra chair hat on), so checking if there is any third conversation happening as well within k8s-infra group?

@Priyankasaggu11929 This is out of scope for the Infra SIG. so we don't currently have any conversation regarding this within the SIG.

[mfahlandt]

There was a discussion about this in the Meeting of SIG k8s infra.
In this meeting it was pointed out SIG Testing have a similar place called experiments https://github.com/kubernetes/test-infra/tree/master/experiment

I would like to align if this is already a good place to be and if SIG Testing want to be the host or if we want to open up a new central place under SIG ContribEx

[justinsb]

So I think that I and @ameukam are happy to start the ball rolling under kubernetes-sigs/maintainers, and I think we have consensus that this works as a venue, under an "experiments" folder.

For now then we don't need a WG or a separate repo.

Let's keep iterating under kubernetes-sigs/maintainers in the experiments folder, see what projects come out of it and who gets involved, and then we can use that to make an informed judgment on a WG or repo in a few months?

[ameukam]

@Priyankasaggu11929 For things out of scope at the moment:

• automate away coding using AI tools

[Priyankasaggu11929]

So I think that I and @ameukam are happy to start the ball rolling under kubernetes-sigs/maintainers, and I think we have consensus that this works as a venue, under an "experiments" folder.

For now then we don't need a WG or a separate repo.

Thanks for the confirmation @justinsb.

Let's keep iterating under kubernetes-sigs/maintainers in the experiments folder, see what projects come out of it and who gets involved, and then we can use that to make an informed judgment on a WG or repo in a few months?

Let's document both (i) some timeline to review the progress for the future informed decision + (ii) out of scope items, in a readme in the new experimental/ directory in k-sigs/maintainers.

---

There was a discussion about this in the Meeting of SIG k8s infra.
In this meeting it was pointed out SIG Testing have a similar place called experiments https://github.com/kubernetes/test-infra/tree/master/experiment

Also @ameukam, @justinsb, checking if you both considered the experiment folder in k/test-infra as well?

[ameukam]

Also @ameukam, @justinsb, checking if you both considered the experiment folder in k/test-infra as well?

@Priyankasaggu11929 there is no significant value to use k/test-infra repo at the moment. We can always reconsider its use if necessary.

[BenTheElder]

For now then we don't need a WG or a separate repo.

+1, not in the experimental stage. If we build something promising with increasing scope we should consider spinning it out (like we have done previously e.g. kind, prow, boskos)

@Priyankasaggu11929 there is no significant value to use k/test-infra repo at the moment. We can always reconsider its use if necessary.

I spoke to justinsb about it in the SIG Testing meeting, it has the benefit of an existing repo happy to host these and grant OWNERS to self-approve a subdir doing whatever project automation you want.

Project automation is explicitly in scope for the SIG Testing charter and most project/github automation tools are SIG Testing sponsored (e.g. peribolos, prow, ...), though always in partnership with Contribex especially with respect to policies (which is also clearly denoted in the charter).

I think the maintainers repo makes sense if this is focused around OWNERS automation, otherwise I don't think it's a good fit. I don't think e.g. general review automation makes sense there.

Please FYI SIG Testing and Infra on any new automation we think the project will be running/hosting, if only for awareness to dedupe projects and to consider resource management.

cc @kubernetes/sig-testing-leads

There is another real downside to the maintainers repo: this currently has a single purpose-built tool that should be hosting tagged releases for that tool. I see a request to gain admin permissions over this repo which suggests it's not a simple experimental directory and will include modifying the repo itself. (Versus test-infra is already a catch-all repo for automation+testing configs and tools)

[BenTheElder]

If we need admin permissions then that's a big signal that we need a dedicated single-purpose experiment repo, which is not something the project has done well yet. https://github.com/kubernetes/org/pull/5394/files

If we need that I think we can, but it probably shouldn't be a single-catch all then.

Also, the SIG Contribex Charter does not currently clearly have scope for general automation other than moderation, policies procedure and public platform management.

https://github.com/kubernetes/community/blob/master/sig-contributor-experience/charter.md

[BenTheElder]

I suppose you can argue this is "public platform management", but currently "repository automation tools" are definitely SIG Testing (see: peribolos and the charters).

Regardless, I think the decision tree should be:

• no repo admin permission is needed, just a code folder for some repo automation experiments
  • => experiments/ai-tools/OWNERS in test-infra (and NOT in maintainer-tools, we will make this currently neat little repo bloated otherwise)
• repo admin permission is needed, including releasing binaries etc
  • => dedicated single-purpose repo

[BenTheElder]

/sig testing

[BenTheElder]

(Also re: admin access as a ref flag: not so much for security as we know these contributors, but because admin access == modifying repo global state == not a good candidate for sharing a repo, and also isn't a repeatable pattern for other similar experiments)

[ameukam]

@kubernetes/sig-contributor-experience-leads With Ben's feedback (with steering hat ?) specifically with respect of the SIG Testing charter, Can we proceed with proceed with repo creation ?

[BenTheElder]

@ameukam clarified that admin access is for hosting content on GHCR / publishing releases, which I think indicates that we should make a kubernetes-sigs/github-ai or something (... probably a better name than that) instead of trying to use an experiment directory in any repo.

Also

kubernetes-sigs/maintainers seems to be a nice option then. Both tools (maintainers and the one you are proposing) have similar reasons to exist.

This project seems to have scope far beyond scanning OWNERS files

This tool will leverage different AI tools to automate and enhance various aspects of repository maintenance, such as issue triaging, pull request review, and possibly documentation generation.

Docs generation should also involve looping in SIG Docs, if that's in scope.

If not, let's scope to repo automation / triage and start a repo creation request in kubernetes/org.
I am happy to +1 this effort, IIRC @vllry had pretty good success in the past using a simple model to independently sig-label issues in kubernetes/kubernetes but we never got an official upstream project for this sort of thing.

[Priyankasaggu11929]

@kubernetes/sig-contributor-experience-leads With Ben's feedback (with steering hat ?) specifically with respect of the SIG Testing charter, Can we proceed with proceed with repo creation ?

Let's use one of the upcoming SIG ContribEx meetings to discuss this further and then only we, move ahead with any next steps here.

Since, the supposed scope of this AI project is also going to cross/overalap with the charter of SIG Testing, let's have the discussion with @kubernetes/sig-testing-leads in loop.

@BenTheElder, @ameukam, @justinsb – does tomorrow's ContribEx meeting slot (wed, Feb 12, 9:00 AM PT) work for you all?
(I'll join too if everyone's available and we're scheduling it on tomorrow's agenda. Thanks!)

[ameukam]

SGTM. Can't guarantee I'll join but I'll do my best.

[palnabarun]

If we need admin permissions then that's a big signal that we need a dedicated single-purpose experiment repo, which is not something the project has done well yet. https://github.com/kubernetes/org/pull/5394/files

I am -1 to the admin access request for k-sigs/maintainers anyway. I don't think it's needed.

---

With the added context from @BenTheElder, is SIG Testing then going to primarily own the new repository?

ContribEx and Docs can co-own but with the clarified context for the repo, my opinion is that it's suitable to be owned by SIG Testing.

@kubernetes/sig-testing-leads -- please chime in for an acknowledgement.

[justinsb]

I don't think we need admin access to a GitHub repo - certainly not at the current time.

I think the priority should be to gather code contributions so we can better understand the size and shape of the effort to improve the maintainer experience. For now, let's continue working in the maintainers repo. We can easily redirect to any other repo, all we need is an OWNERs file with @ameukam and myself.