Implement retry on checkout for reusable-ubuntu #129241
Conversation
diegorusso
requested review from
ezio-melotti,
hugovk and
AA-Turner
as code owners
| @@ -34,9 +34,12 @@ jobs: | |||
| PYTHONSTRICTEXTENSIONBUILD: 1 | |||
| TERM: linux | |||
| steps: | |||
| - uses: actions/checkout@v4 | |||
| - uses: Wandalen/wretry.action@master | |||
There was a problem hiding this comment.
Is there a feature request for actions/checkout to retry checkouts?
I'm unsure about using a third party action here, vs e.g. a run: block just using git commands.
There was a problem hiding this comment.
We raised a support ticket with GitHub and a couple of tips have been generated. One of this was to add a retry to the checkout.
Funny enough, the retry doesn't exist on the checkout action!
I'm going to raise one
There was a problem hiding this comment.
Raised: actions/checkout#2048
There was a problem hiding this comment.
Re: https://github.com/diegorusso/cpython/actions/runs/12948283932/job/36116629838#step:4:157
That's just the third-party linter that complaining about something that either doesn't exist or hasn't been added to the linter's schema.
Please could you rename your branch to 3.14-test-retry so it will trigger some jobs that use actions/checkout?
There was a problem hiding this comment.
Same thing: https://github.com/diegorusso/cpython/actions/runs/12949425398
BTW also the GitHub editor was complaining about the retry.
There was a problem hiding this comment.
Thanks, and there it is on test.yml:
Invalid workflow file: .github/workflows/build.yml#L258
The workflow is not valid. In .github/workflows/build.yml (Line: 258, Col: 11): Error from called workflow diegorusso/cpython@6a298de (Line: 40, Col: 7): Unexpected value 'retry'
https://github.com/diegorusso/cpython/actions/runs/12949425427
It does look like actions/checkout does have a non-configurable 3-attempt retry already built-in:
There was a problem hiding this comment.
I think here we are looking at this from the wrong angle. If you see the failures posted initially, they don't fail while checking out the repo.
It is not started at all. After it prints the env variable there is nothing. I have the feeling that there is some infrastructure issue in running that action on a specific node.
Even if we put the retry 100 times, they won't be executed as this fails even before to start the process.
There was a problem hiding this comment.
Yes, that's the impression I get too.
Looking more closely at the raw logs (https://github.com/python/cpython/actions/runs/12934438169/job/36075486419?pr=129232 > ⚙️ > View raw logs), the checkout does actually succeed.
For some reason, the next step just doesn't start.
Comparing with a passing build:
(Not shown in the diff, the last line of the failing build is:)
2025-01-23T17:13:58.8145851Z Cleaning up orphan processes
There was a problem hiding this comment.
Oh wow, I didn't remember that we had access to these raw logs. They are useful indeed.
So the retry is pointless because it actually doesn't fail on the checkout.
Do you know what the actual error is here? There's not much in the logs. |
Not yet. @ambv has enabled debug information so next time that happens hopefully we will know what is going on. |
I have a 404 on the link. |
There are a few occasions where the checkout on reusable-ubuntu is failing. This happens on Ubuntu arm for free threading builds. Examples:
The retry mechanism should protect us from a failure.