Support Python 3.12 on Cygwin #1988
Conversation
EliahKagan
force-pushed
the
cygwin-py312
branch
from
052a7ed to
3dc5366
Compare
|
In both this older run before the #1989-related rebase, and this newer run, the Cygwin Python 3.12 job seems to stall either when attempting to download This resembles the problem described for Cygwin I have not reproduced that problem locally. The problems I encounter locally--other than the venv Edit: I also tried locally running the tests with an MSYS2 build of Python 3.12. (That is, with the downstream Python 3.12 packaged by the MSYS2 project for the MSYS environment. Other environments are native Windows builds, while the MSYS environment is Cygwin-like due to MSYS and MSYS2 being forked from Cygwin. In the MSYS2 build of Python, |
Python 3.12 is currently marked "test" and should not be expected to fully work. Assuming this successfully installs it, based on local testing I expect two encounter two problems: 1. Currently, creating a venv with `pip` in Python 3.12 on Cygwin does not seem to be working, even though its own global `pip` exists. Running `python -m ensurepip` likewise does not work, reporting that the bundled `pip-24.3.1-py3-none-any.whl` does not exist. The https://bootstrap.pypa.io/get-pip.py script can be used as a workaround, assuming the problem also happens on CI (which I expect). Eventually, `pip` should be fully working. 2. Once that is worked around, there seem to be problems where Python processes terminate unexpectedly and wrongly report success, or where subprocess creation fails. When I ran `pytest` loading it as a module (`python -m pytest`), it collected tests and sometimes started running them, but then suddenly terminated with an exit status of 0. When I ran it via the `pytest` command (no `python -m`), that problem also usually happened, but one time I got an immediate termination instead, reporting the following errors, yet still with a 0 exit status: 0 [main] python3.12 2724 C:\cygwin64\bin\python3.12.exe: *** fatal error in forked process - WFSO timed out performed fork fixups and dynamic dll loading 0 [main] python3.12 2769 C:\cygwin64\bin\python3.12.exe: *** fatal error in forked process - WFSO timed out performed fork fixups and dynamic dll loading And one time I got an immediate termination reporting this slightly different error, also with an exit status of 0: 0 [main] python3.12 3371 C:\cygwin64\bin\python3.12.exe: *** fatal error in forked process - WFSO timed out after longjmp This curious combination of errors (or the same error arising and being reported in different ways?) is my main motivation for testing GitPython on CI with Python 3.12 in Cygwin now, even though the Cygwin package `python312` and assocated packages are still marked "test". If the problems I observed locally can be reproduced, then this may help to find a minimal case that shows the problem (assuming that it is due to a Cygwin-related bug). This commit contains no attempt to avoid or work around either of those two anticipated problems.
This works around a problem in Python 3.12 on Cygwin where `pip` fails to intall in a venv via `ensurepip` (both implicitly when the venv is created normally, and explicitly when `python -m ensurepip` is attempted), by creating the venv with `--without-pip` and then, once the venv is set up, bootstrapping `pip` in it by running the https://bootstrap.pypa.io/get-pip.py script. Eventually, when `pip` is working automatically in a venv in Python 3.12 on Cygwin (the problem is specific to that combination, and the Cygwin `python312*` packages are still marked "test"), this special-casing can be removed.
EliahKagan
force-pushed
the
cygwin-py312
branch
from
3dc5366 to
399eb7d
Compare
|
The stall still happens, both on CI and locally, including after upgrading from 3.12.8-1 to 3.12.8-2. I believe it is known still to occur, having been reported upstream by others, including for this version--I'm subscribed to the mailing list and I vaguely recall this to be so--but I'll try to check. It resembles the original problem that keeps 3.9 on Cygwin from having a stable version higher than 3.9.16. The stall is not specific to installing mypy--it will occur when installing other dependencies. Because I have a substantial backlog of more important things than this PR, both in GitPython and elsewhere, my plan right now is to check if the 3.9 job itself is still working on the main branch and, if so, to close this PR, with the hope of revisiting it in the future if bugs in the Cygwin downstream Python 3.12 package are fixed. Edit: The Cygwin test job is broken on the main branch as well. See #2004 for details. These are very unlikely to share a common cause, and a fix for that seems very unlikely to affect this in any way. Nonetheless, I'll keep this open for a short while longer or until that is fixed. This is in the hope of being able to confirm, when closing this without merging, that the matrix is capable of generating a working job (for 3.9) but that 3.12 is still not working Over time, my guess is that this will get fixed, but if not, Python 3.12 on MSYS2 does noth have similar problems. So I predict that, if this does not end up being fixed in the Cygwin packages, then some current users of Python on Cygwin will use it on MSYS2 instead. If that becomes popular among GitPython users, then an MSYS2 job should be added. Right now, though, that's all hypothetical; unless requested by users, I don't think it's currently a priority to test GitPython on the MSYS2 downstream build of Python. Relating to MSYS2 environments, only the MSYS environment is Cygwin-like rather than native. The MINGW64, UCRT64, and CLANG64 builds of Python installable through the MSYS2 package manager--unlike what I am calling the MSYS2 builds, i.e. unlike the MSYS builds for MSYS2--are native builds, and I expect them already to work. The CLANGARM64 build, which I haven't worked with, is likewise native and comparable to upstream/normal ARM64 builds, which has stable versions on the downloads page. We don't currently test AArch64/ARM64 at all on Windows. That may be something to look into once GitHub-hosted ARM64 Windows GHA runners are available without fee. From past cloud experiments on ARM64 Linux, and past QEMU experiments with the even more different s390x architecture--taken together with how the macOS jobs run on Apple Silicon, which is ARM64--my guess is that GitPython will behave the same and have no new problems on ARM64 Windows compared to x86-64 Windows. But it might still be good to try testing more than one architecture on Windows. However, because GitPython and its gitdb and smmap dependencies, like many Python projects, are mostly written in a way that abstracts out details that vary between architectures, it seems to me that testing on a variety of architectures is lower priority in GitPython than in some other projects such as gitoxide. |
|
I don't know if it's related, but Cygwin seems broken on |
|
The Cygwin breakage on main is due to #2004. I don't think there's any connection between this and that, but that issue is why I haven't closed this yet (per #1988 (comment), I want to verify that Python 3.12 is not working even when Python 3.9 is working). The reason #2004 wasn't observed on main earlier is just that there was no event to trigger a run; I did produce it on the main branch of my fork. I do very much expect that #2004 can be solved (it looks nontrivial but not nearly as hard as the issue here). |
|
Once CI is green again, such as after #2009, I can try to rebase this to verify that it is really not working due to problems we can't fix here (before closing it). |
Status - it's not workingI haven't rebased this, but Some new package versions have been released. That happens both before and after upgrading to them. The above run is done with the newer versions (which are shown in a log below). An occasional message that may or may not be relevantOccasionally I get: I used an absolute path there. The messages far more often are shown when using an absolute path, and possibly are never shown when just using Producing
|
Cygwin has had Python 3.9 as its latest version for a while, with progress both on moving past 3.9.16 for Python 3.9, as well as on packaging Python 3.y for higher y, stalled due to the problem described in #1814. However, packaging work has resumed. This includes Cygwin builds of Python 3.12 (which is successfully packaged for the parallel/forked MSYS2 target). Such builds are marked "test" and do not work properly yet (reports and discussion can be seen in this month's list archive).
Since GitPython has long-standing support for Cygwin, which I suspect remains one of the important use cases of GitPython (with some alternatives not supporting Cygwin), it seems to me that it is worthwhile to support Python 3.12 on Cygwin, as well as to test it on CI. Since the Cygwin downstream build of Python 3.12 is currently experimental and not fully working, I don't expect GitPython to work with it yet. This PR is for testing that, and for keeping track of changes that are known to be wanted whenever Cygwin Python 3.12 is working. I have encountered substantial differences between CI and my local setup with Cygwin in the past, so testing this on CI now may help avoid surprises. It may also lead to the discovery of bugs, or information about known bugs, that could be useful to report on the Cygwin mailing list. This PR is definitely not ready to be merged.
This PR is separate from and in parallel to #1955 (which is about Python 3.13 and blocked due to a need to change some GitPython code to support that version on Windows), because they are conceptually separate with no known overlap in blockers or in changes that would fix them, and because it seems to me that having two PRs makes the situation clear, both to me and to other people who might wish to contribute.