AI patch summary MVP

[JelteF]

One thing AI is actually very good at these days is reading a bunch of text very fast and producing a pretty useful (and mostly correct) summary. One specific type of summary this could give is what the blockers on a current patchset are and what are the disagreements. Such things for a very long email thread.

Sadly doing this automatically is no small task for at least a few reasons (and probably more):

1. Who pays for the AI stuff?
2. When is it run?
3. Which AI tool to use?
4. Most AI tools cannot read contents of links that you provide it
5. The commitfest app does not actually store any data.
6. There are hard limits on the amount of text you can give to an AI to summarize.

So instead of creating a solution that's fully integrated in the commitfest app, it seems better to start with an MVP that allows us to evaluate the usefulness of such summaries.

An MVP that should be relatively simple to implement is having a button on the patch page that populates the clipboard with a prompt that people can copy paste in their favorite AI tool. This would work around the first three issues. To solve the remaining issues you'd have to:

1. Include the actual email thread text (and maybe patch text)
2. Allow limiting the text in the prompt to a certain character limit (probably with a default that's allowed for the most popular AI tools)

[destrex271]

This will be a super helpful feature!!

Which AI tool to use?

I have two temporary/placeholder solutions to get started with this MVP:

• We can utilize the Google Gemini API to spin up a quick MVP
• We can use lightweight models from ollama like 'phi-3' and maybe setup a test instance for this? This will be a little harder to setup but might give us some pointers as to how we can host these models better

Most AI tools cannot read contents of links that you provide it

We have the content on our mail archives, so maybe we can either use web scraping or expose an endpoint from the archives website(or from wherever we get the email data) and feed that data to our model

There are hard limits on the amount of text you can give to an AI to summarize.

For the MVP I think we should focus on why the patch was required and who is involved with it. For example: Patch A was written because of bug x, and these people are involved in this. We can expand on the other components like what, when etc.. later.
Correct me if I am wrong, but I think we can get this information from first 1-2 emails in the threads associated with a patch for now.
Also maybe we can limit the scope of this feature to patches tagged as Ready for Commiter(or any other category that might help us in evaluating the usefulness of this feature)

If this seems reasonable, I can start working on small prototype with some dummy data.

[JelteF]

This will be a super helpful feature!!

Which AI tool to use?

I have two temporary/placeholder solutions to get started with this MVP:

* We can utilize the Google Gemini API to spin up a quick MVP

* We can use lightweight models from ollama like 'phi-3' and maybe setup a test instance for this? This will be a little harder to setup but might give us some pointers as to how we can host these models better

I definitely think that we'll eventually want do something like this. But the first option means we need to set up an account and figure out who pays for it. The second one means significant infrastructure work. As explained in my initial description, I think for an MVP we could skip these and only have a "populate clipboard with AI summary prompt" button. But if you think option 1 (gemini API) is not a lot of work to set up, feel free to include that too. As long as it's optional (based on an API key being being present in the settings) I think that's fine.

Most AI tools cannot read contents of links that you provide it

We have the content on our mail archives, so maybe we can either use web scraping or expose an endpoint from the archives website(or from wherever we get the email data) and feed that data to our model

I think scraping is probably the easiest. You can go to a whole thread by going to: https://www.postgresql.org/message-id/flat/

There are hard limits on the amount of text you can give to an AI to summarize.

For the MVP I think we should focus on why the patch was required and who is involved with it. For example: Patch A was written because of bug x, and these people are involved in this. We can expand on the other components like what, when etc.. later. Correct me if I am wrong, but I think we can get this information from first 1-2 emails in the threads associated with a patch for now. Also maybe we can limit the scope of this feature to patches tagged as Ready for Commiter(or any other category that might help us in evaluating the usefulness of this feature)

If this seems reasonable, I can start working on small prototype with some dummy data.

I'd definitely want to have it include some of most recent emails too. Threads tend to change focus a lot over time. And I think one of the most useful things is to figure out the current status of the thread, and what it's blocked on.

[destrex271]

I'd definitely want to have it include some of most recent emails too. Threads tend to change focus a lot over time. And I think one of the most useful things is to figure out the current status of the thread, and what it's blocked on.

Yes this information is much more useful.

But if you think option 1 (gemini API) is not a lot of work to set up, feel free to include that too.

It's not a lot of work to setup but I'll need to use the generative-ai(https://pypi.org/project/google-generativeai/) sdk by google to access Gemini.(I can just send http requests, but that will be a lot of stuff to setup)

For now I think we can experiment with the "populate clipboard with AI summary prompt" button option for threads with relatively less number of threads.

I'll try to set it up on a different branch in my repository asap and maybe we can see how it works out.

Once we have this working, we can try sending it to Gemini and see what it generates.