CI: Azure_Tests Win timeout occasionally

[PardhavMaradani]

Expected behavior

Tests don't timeout and complete

Actual behavior

This was observed only on a couple of PR runs so far, but could become a potential issue going forward.

• Failure run 1
• Failure run 2

A few notes about these runs:

• They are from different PRs, so the issue is not related to code or actual tests
• Here is a successful run after an empty commit after the Failure run 1 above - indicating that this could potentially be an infra issue
• Successful runs for the Run MDAnalysis Test Suite job take around 40 mins
• Failure runs for the same job take over 50mins and the tests are only around 75% done (triggering the overall 1 hour timeout)

Code to reproduce the behavior

So far, the failures are only a couple and a bit random

[IAlibay]

Those tests shouldn't be taking so long.. it looks like something recently significantly increased our test times, will need to spend some time digging into it.

[PardhavMaradani]

I didn't notice a significant jump at least from when Azure_Tests Win-Python313-64bit-full-wheel was added (~39m then to ~42m now), but in both the timed out cases so far, the workers seem to be consistently slower as seen below:

	Works 1	Works 2	Fail 1	Fail 2	Fail 3
Initialize job	3s	4s	4s	6s	5s
Checkout MDAnalysis	56s	1m 16s	1m 45s	2m 0s	2m 10s
Use PythonVersion	<1s	<1s	1s	1s	2s
Build MDAnalysis (wheel)	2m 38s	3m 3s	4m 36s	5m 7s	5m 18s
Install tools	4s	8s	12s	12s	14s
Install dependencies	28s	21s	30s	34s	34s
pin to older NumPy	10s	11s	17s	20s	19s
Install additional deps	38s	46s	1m 7s	1m 10s	1m 16s
List of installed deps	2s	3s	4s	3s	4s
Build MDAnalysis	<1s	<1s	<1s	<1s	<1s
Check installed packages	2s	4s	4s	4s	5s
Run MDAnalysis Test Suite	36m 56s	36m 6s	51m 30s X	50m 31s X	49m 55s X

There are only three failed runs, so too few data points. Thanks

[IAlibay]

Thanks for the timing breakdown, I'll look into it later - for now at least it's only an occasional fail.

[PardhavMaradani]

There were a couple more failures since. The reason we are seeing this now could also be potentially due to the switch over to 'windows-latest' (PR #5167) as described here - not sure. Thanks

[yuxuanzhuang]

Actually, even the tests that passed include timeouts...somehow the timeouts aren’t being reported as failures.

https://github.com/MDAnalysis/mdanalysis/actions/runs/20822770270/job/59815655917#step:8:398

[yuxuanzhuang]

https://github.com/MDAnalysis/mdanalysis/actions?page=42

Something was broken between Nov.6 (20min) and Nov.9 (40min timeout).

Take that back, actually, the timeout already happens in the ~20 min passed tests. Just somehow passed in a shorter time. The older actions were deleted so it's hard to pinpoint when the timeout started to happen.

https://github.com/MDAnalysis/mdanalysis/actions/runs/18784817388/job/53600207681

Investigat-ing

[tylerjereddy]

If it becomes more than a little annoying, just moving a few windows jobs over to GitHub Actions CI may be the best bet. I believe the original Azure CI additions were from me, mostly because I already did the same work upstream for NumPy and SciPy. They've both since abandoned Azure because it eventually became problematic in similar ways, and GHA supports the necessary platforms now.

[yuxuanzhuang]

It’s not just Azure that’s timing out, other Linux CI jobs are hitting the same issue. So I think we can live with it for another day.

What’s happening is that the multiprocessing tests can deadlock because each xdist worker creates a multiprocessing.Pool using the default fork start method. I’ve updated the tests to use spawn instead. (I believe I tried this the last time we saw timeouts; it didn’t address the deeper GSD issue at the time, so we didn’t merge it.)

What’s weird is that Windows should default to spawn, but it still seems to hang. My guess is that the previous implementation (not using a context manager) wasn’t cleaning up the pool, so the test could get stuck during join. I’m not able to read the failed Azure runs because the logs were cleaned up, but I’ve tested it twice locally and it seems to be working for now.