CI is flaky

[mkantor]

I've noticed recently test runs in CI intermittently fail during the samples_match_snapshots test. Here are some examples:

• https://github.com/mkantor/operator/actions/runs/15970415912/job/45040169655
• https://github.com/mkantor/operator/actions/runs/15997922796/job/45125542990
• https://github.com/mkantor/operator/actions/runs/16000043077/job/45132410874
• https://github.com/mkantor/operator/actions/runs/16024351981/job/45208540283
• https://github.com/mkantor/operator/actions/runs/16024739305/job/45209819465
• https://github.com/mkantor/operator/actions/runs/16567584490/job/46851298546
• https://github.com/mkantor/operator/actions/runs/16598961163/job/46952948227

The error message is as follows:

thread 'samples_match_snapshots' panicked at tests/lib/mod.rs:330:36:
Failed while sending request for […]: Failed to connect to host: Internal error: connector has been disconnected
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Panic in Arbiter thread.

The specific example in the error message is not consistent across failures.

I haven't done any deeper investigation, but it smells like a timeout or resource limit or somesuch. Perhaps there was a configuration change in GitHub Actions which caused this to start happening.

[mkantor]

It seems like this happens consistently when running CI on the master branch after merging a pull request, and not (ever?) when running the same CI job against the same exact code on other branches before merge. It also seems like a single re-run of the failed job reliably succeeds. I should keep an eye out for counterexamples to make sure this isn't a red herring.

It's not immediately obvious why the branch name should be at all relevant. The only thing that comes to mind is that maybe GitHub is using a different pool of runners for the master jobs, and they need to be warmed up first?

[mkantor]

I just hit this in the CI run on a pull request (I think this is the first time that's happened):

• https://github.com/mkantor/operator/actions/runs/16596739169/job/46945143280

And it worked upon re-run, just like I've seen with the other failed jobs.

Interestingly, this same commit also worked on the first run after merging to the master branch.