The "Python package" workflow's pytest job has been crashing on main for over a month - every run I can find back to June 4 is red (gh run list -R openclimatefix/graph_weather --branch main, filtered to the "Python package" workflow, is failure/failure/failure/... all the way back).
It's not individual test failures - pytest itself aborts during collection:
- Ubuntu:
Fatal Python error: Segmentation fault (exit code 139)
- macOS:
libc++abi: terminating due to uncaught exception of type std::length_error: vector then Fatal Python error: Aborted (exit code 134)
Both tracebacks bottom out in the same place:
File ".../site-packages/torch/_ops.py", line 1442 in load_library
File ".../site-packages/torch_scatter/__init__.py", line 16 in <module>
File ".../site-packages/torch_geometric/typing.py", line 139 in <module>
torch_geometric/typing.py tries to load torch_scatter's native extension via ctypes/torch.ops.load_library, and that call crashes outright. This happens as a side effect of collecting whichever test file pytest happens to reach first that imports graph_weather.models (I saw it triggered via both tests/test_fgn.py and tests/models/layers/test_efficient_batching.py in different runs) - it's not particular to either file, it's the torch/torch_geometric/torch_scatter combination in the pixi-resolved environment that's broken.
Given torch 2.10.0 shows up in the logs and the pixi manifest pins torch_geometric = "*"/installs torch_scatter via the installpyg task against torch-2.7.0+cpu wheels, my guess is the pixi env is now resolving a torch_scatter build compiled against a different torch ABI than the torch that's actually installed (2.10.0). That would produce exactly this kind of native crash on first load. I haven't been able to pin down the exact fix since I don't have the pixi-managed environment easy to reproduce outside CI, but the version skew between the installpyg/installpygcuda tasks and the actual pinned torch version seems like the place to start looking.
This means every PR's pytest check is currently failing regardless of what the PR actually changes - ran into it on #230 and #231, neither of which touch anything downstream of torch_geometric.
Recent failing runs on main for reference (all "Python package" workflow, pytest jobs):
It's been failing this way on every push to main back to at least June 4 (26931347935), so this isn't a one-off flake.
The "Python package" workflow's
pytestjob has been crashing onmainfor over a month - every run I can find back to June 4 is red (gh run list -R openclimatefix/graph_weather --branch main, filtered to the "Python package" workflow, is failure/failure/failure/... all the way back).It's not individual test failures - pytest itself aborts during collection:
Fatal Python error: Segmentation fault(exit code 139)libc++abi: terminating due to uncaught exception of type std::length_error: vectorthenFatal Python error: Aborted(exit code 134)Both tracebacks bottom out in the same place:
torch_geometric/typing.pytries to loadtorch_scatter's native extension viactypes/torch.ops.load_library, and that call crashes outright. This happens as a side effect of collecting whichever test file pytest happens to reach first that importsgraph_weather.models(I saw it triggered via bothtests/test_fgn.pyandtests/models/layers/test_efficient_batching.pyin different runs) - it's not particular to either file, it's thetorch/torch_geometric/torch_scattercombination in the pixi-resolved environment that's broken.Given torch 2.10.0 shows up in the logs and the pixi manifest pins
torch_geometric = "*"/installstorch_scattervia theinstallpygtask againsttorch-2.7.0+cpuwheels, my guess is the pixi env is now resolving atorch_scatterbuild compiled against a different torch ABI than thetorchthat's actually installed (2.10.0). That would produce exactly this kind of native crash on first load. I haven't been able to pin down the exact fix since I don't have the pixi-managed environment easy to reproduce outside CI, but the version skew between theinstallpyg/installpygcudatasks and the actual pinned torch version seems like the place to start looking.This means every PR's
pytestcheck is currently failing regardless of what the PR actually changes - ran into it on #230 and #231, neither of which touch anything downstream oftorch_geometric.Recent failing runs on
mainfor reference (all "Python package" workflow,pytestjobs):It's been failing this way on every push to
mainback to at least June 4 (26931347935), so this isn't a one-off flake.