Platform Tag Improperly Set for Mac OS Wheels

[michaeltryby]

Great work Henry! I'm migrating a project from scikit-build to scikit-build-core and have found the process straight forward.

I do have a suggestion though ...

Currently, the only way to set the cmake variable "MACOSX_DEPLOYMENT_TARGET" is via environment variable. Attempts to pass it as an argument to cmake or set it within a CMakeLists.txt file are clobbered by the default to the current MacOSX version. This logic is easily observed in the code and straightforward to reproduce.

The behavior, however, seems counterintuitive to CMake variable setting presedence which goes something like ...

passed command line argument > setting in list file > environment variable > default

Please consider updating this behavior.

Keep up the great work!

[LecrisUT]

Were you using CMAKE_OSX_DEPLOYMENT_TARGET and something along the lines of this documentation page?

CMake's documentation on MACOSX_DEPLOYMENT_TARGET says that it is only used as a default variable for CMAKE_OSX_DEPLOYMENT_TARGET.

Also, out of curiosity, where do you use this? Shouldn't the CI system set this accordingly? cibuildwheel seems to have an integration with that ¹.

Footnotes

1. https://github.com/search?q=repo%3Apypa%2Fcibuildwheel%20MACOSX_DEPLOYMENT_TARGET&type=code ↩

[henryiii]

Besides cibuildwheel, conda-build/rattler also sets it. If you are not building redistributable wheels, you generally don't need to set it at all, and if you are, you should be using something like cibuildwheel that can build redistributable wheels, this is one part of that. Usually you should set it via environment variable, so that all dependencies also build with the correct deployment target. Setting it manually would potentially not build everything with the correct target.

If you set it in pyproject.toml, how would you handle conflicts with cibuildwheel and other build environments? You can already set CMAKE_OSX_DEPLOYMENT_TARGET in pyproject.toml, though I'm not sure we check that for the wheel name - we likely override it if the envvar is set.

(And if you are using cibuildwheel, you can set this in pyproject.toml for cibuildwheel already)

[michaeltryby]

Hey guys! Thanks for the quick response.

Correct. I should clarify. When CMAKE_OSX_DEPLOYMENT_TARGET is defined in the pyproject.toml file as a cmake.define or set directly in the CMakeLists.txt file it gets clobbered.

I'm wrapping a library with no dependencies that require relocation. As such cibuildwheel is overkill for my use case. Have either of you considered scikit-build-core running outside cibuildwheel? Shouldn't that be your primary use case?

I'm comfortable with cmake and find sckikit-build-core behavior here counterintuitive. Clobbering defines and list file settings isn't the expected behavior.

As a workaround hatch can be configured to set MACOSX_DEPLOYMENT_TARGET in the default environment.

Thanks again and keep up the great work!

[LecrisUT]

it gets clobbered

Not sure what you mean here. It gets removed? We don't alter CMAKE_OSX_DEPLOYMENT_TARGET afaict, so if you want to manually override it, you should be able to. Otherwise it would indeed seem like a bug.

Have either of you considered scikit-build-core running outside cibuildwheel? Shouldn't that be your primary use case?

For me, CI wise, I have one project that I test against Github Actions, conda and Fedora. But after CD, I only test them within Fedora because I don't know of tooling to test those other than cibuildwheel.

[henryiii]

My guess is "clobbered" means that when it computes the wheel tag it doesn't look up CMAKE_OSX_DEPLOYMENT_TARGET but just sets the current macOS version as the tag. If you set it in your CMakeFile, it is not reasonable to look it up, though it can be done if it's set in your pyproject.toml, but that seems a little weird that they would not be the same. And you'd have to look in both the cmake.defines and parse the cmake.args.

cibuildwheel does a lot of other things for you too, such as downloading the official macOS installers for Python rather than the GitHub Actions installers¹ (which are compiled for the lowest version GitHub supports, not 10.13), running auditwheel, etc.

We very much support running outside of cibuildwheel if you don't want to redistribute, but for redistribution, we require you use a tool that supports redistribution, like cibuildwheel, multibuild, conda-build, rattler, or any distributions (pyodide, homebrew, fedora, Debian, arch, etc).

Footnotes

1. GitHub Actions does now use the official installers for Apple Silicon, but not Intel ↩

[michaeltryby]

No problem. When I say "clobbered' I mean that the setting I'm passing is being ignored or overwritten.

Right on. I build on Github Actions for Windows, Linux, and MacOS. Shout out to dockcross! It's a great tool that I use to build, repair, and test manylinux wheels using a slim 55 line bash script.

In my experience those tools aren't required for redistributable wheels.

[henryiii]

Windows is easy, nothing special is required. :)

To be clear, all any of these tools are doing is setting MACOSX_DEPLOYMENT_TARGET in the environment. And CMake supports this. And hard coding this to some value (say, 10.13) instead of setting an environment variable in your CI will mean that someone building your wheel directly will not get a native wheel when they build locally, meaning they have to pay the price of the compatibility even if they don't need it. I'd say hard-coding it, such as in your CMakeLists, is an anti-pattern.

Anyway, I wonder if we could inspect CMakeCache.txt for CMAKE_OSX_DEPLOYMENT_TARGET and use that if it's cached, as we don't need the wheel name until after it is configured/built. I don't remember if there's a place we are required this to be statically computable (true for much of the metadata, but I think maybe not the wheel name).

[LecrisUT]

My guess is "clobbered" means that when it computes the wheel tag it doesn't look up CMAKE_OSX_DEPLOYMENT_TARGET but just sets the current macOS version as the tag.

Ok, so if my understanding is correct, and @michaeltryby hope you can confirm as well. The issue is not on CMake build phase, i.e. it should work fine when run as pip install -e. But the issue is when scikit-build-core creates the .whl archive because the tag is not consistent there and pip is trying to consume it? Couldn't you manually fudge it with wheel.py-api to confirm the issue?

In my experience those tools aren't required for redistributable wheels.

Once you get into the nitty-gritty and publish on PyPI for all linux users, then the gnarly issues with C compilers, standards, etc. pop up. Otherwise power to you to use your favorite build process, it helps us find these edge cases ;)

Windows is easy, nothing special is required. :)

The lack of RPATH in windows is always killing me 😱

Anyway, I wonder if we could inspect CMakeCache.txt for CMAKE_OSX_DEPLOYMENT_TARGET and use that if it's cached, as we don't need the wheel name until after it is configured/built.

Yeah makes sense. Let's hope somebody doesn't set it with set() later on 😬. But the documentation explicitly says not to do that :D

[michaeltryby]

Thanks! Keep it up!

[michaeltryby]

Hey Guys! Wanted to add latest observations to the Issue. Hopefully this will help you better diagnose the problem.

When building wheels on an Intel Mac and cross-compiling arm64 using Xcode. I added the following to my pyproject.toml file.

[tool.scikit-build.cmake.define]
CMAKE_OSX_ARCHITECTURES = "arm64"
CMAKE_OSX_DEPLOYMENT_TARGET = "11.0"

I've confirmed that the cmake build has the -arch arm64 and -mmacosx-version-min=11.0 compiler flags set. The platform tag on the wheel, however, reads macosx_15_0_x86_64.whl.

When I install the wheel and run tests they fail because the binaries are in fact being compiled arm64.

E   ImportError: dlopen  ... (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64'))

So it seems that the scikit-build-core scheme for determining the platform tag for Mac builds is not working properly.

Hope this helps! Thanks!

[henryiii]

You have to set this with environment variables to cross-compile. If you hard code it in tool.scikit-build.cmake.define, you would make it impossible to build on Intel natively, such as if someone just wanted to or needed to build from source. The standard variable for this is ARCHFLAGS. See:

https://github.com/pypa/cibuildwheel/blob/b656a829c8a01088b5448cea955a5e441de557ce/cibuildwheel/macos.py#L319-L331

This works on all backends, like setuptools, meson-python, maturin, etc.

(For completeness, here is the macOS version setting too: https://github.com/pypa/cibuildwheel/blob/b656a829c8a01088b5448cea955a5e441de557ce/cibuildwheel/macos.py#L295-L316)

[michaeltryby]

It's completely reasonable to expect build-backend = "scikit_build_core.build" to build a vaild wheel for the arguments being passed to it. The fact that it doesn't is a bug.

[LecrisUT]

The issue is that right now scikit-build-core only reads MACOSX_DEPLOYMENT_TARGET environment variable for figuring out what the .whl file and compiled python module names should be. Indeed the action for this issue is to read CMAKE_OSX_DEPLOYMENT_TARGET as well.

But you pointed out a workaround for you is to set MACOSX_DEPLOYMENT_TARGET in hatch, but is there a problem if you set it in pip install directly?

Also about the current issue, is it a problem when you try to build and install for your current macOS environment? I would think that CMake defaults would be clever enough to build when no extra variables are passed to work for your current environment. The default experience that a user running pip install from the source files is to build and install for the current environment, other configurations should be handled by environment variables, --config-settings flags or tool.scikit-build.overrides.

Otherwise I guess you are trying to cross-compile? In which case is it for your local test system or for deployment like in PyPI? For the latter you should consider making the pyproject.toml as general as possible so that other users can build it as well. In either case, how are you interacting with the build, e.g. via pip/uv, are you passing additional variables, etc.?

For the case of manually defining the variables as

[tool.scikit-build.cmake.define]
CMAKE_OSX_ARCHITECTURES = "arm64"
CMAKE_OSX_DEPLOYMENT_TARGET = "11.0"

it would be a footgun because if you then share your project with somebody running intel macOS, then they would not be able to run that executable either. So we are more weary about blindly reading the variable directly because we do not want to just hand over a footgun, so apologies for the multiple back and forward on this while we iron out how/if this needs to be supported.

[michaeltryby]

Binary distributions like wheels were created so that end users would not have to build packages themselves. Therefore, the primary actor in configuring and building binary distributions has shifted from the end user to the package develeloper. If an end user on a platform without a wheel does build the package, they aren't going to be running cibuildwheel. They are probably going to build it directly on their target platform using scikit-build-core or the backend specified by the developer in the pyproject.toml file.

Among other things, the move away from setup.py to pyproject.toml rationalizes and centralizes build configuration settings. Therefore, an end user building a package themselves should only have to look in one place -- the pyproject.toml file -- to adjust build settings for their target platform.

Your reliance on undeclared dependencies like cibuildwheel and on undocumented environment variables is far more of a "footgun" than using a documented feature of your package -- [tool.scikit-build.cmake.define] -- to declare valid build settings in a logical and editable way.

I understand that you have constraints on how to implement things, however, leaving this broken in not a good design tradeoff.

[henryiii]

The "developer" produces wheels for common platforms. It's fine to assume they follow standards that work across all backends for those - MACOSX_DEPLOYMENT_TARGET is one of the most well documented and standard mechanisms out there, even CMake documents that CMAKE_OSX_DEPLOYMENT_TARGET default to MACOSX_DEPLOYMENT_TARGET. Every build system, not just cibuildwheel, even non-Python ones like homebrew, etc. sets this; they do not need to set CMake flags, because this works with all build systems. Even Meson, which hates environment variables, respects this one. ARCHFLAGS is a bit less common, but it's still a macOS standard.

If a "user" runs pip install <package> and there are no wheels for the platform they are on, or if they are requesting a full source build for some reason, then they want native wheels. They want the current version of macOS they are on and the architecture they are on. They don't have the ability to edit the pyproject.toml - they never even see the pyproject.toml, it's just part of the SDist they pulled with pip (or uv). You don't want to force something else on them via hard-coded settings. If they are doing something advanced and not doing a native build, environment variables are perfect for this sort of thing, since they can't edit the pyproject.toml.

I think its fine to expect scikit-build-core to respect the standard¹ mechanisms for cross-compiling and macOS version targeting like all other build backends for Python, and not be able to read the CMake way of doing things (remember, it can be set via define, by args, in your CMakeLists, or ideally a toolchain file!), even if it would be slightly better to be able to respect both. I think if it's reasonable to add, we can add it, but it's a) a low priority, b) requires writing a parser for CMakeCache.txt (hopefully doess't require parsing a toolchain file!), c) is rather invasive to the build procedure, since computing a tag now requires reading and parsing files produced in an earlier step, and d) might actually encourage bad practices like hardcoding platform assumptions into a pyproject.toml file. On the plus side, though, this might be helpful for linux cross-compiling, where there are no standards currently and it's likely to take a while to get a standard.

Short-term, improving our docs is probably the first thing to work on. https://scikit-build-core.readthedocs.io/en/latest/crosscompile.html does mention ARCHFLAGS, but could also mention MACOSX_DEPLOYMENT_TARGET and could mention things that are not supported. Also, I see we have custom handling for CMAKE_SYSTEM_PROCESSOR in cmake_args (only). I think fixing that to handle all posibilties (and we could go ahead and add handling for at least the macOS version too while at it) could be medium term priority. And I'd be fine to review a PR for even a low priority plan. ;)

If you are cross-compiling, you should also be setting at least CMAKE_SYSTEM_PROCESSOR and CMAKE_CROSSCOMPILING, by the way. Generally, you are supposed to make a toolchain file, though we don't officially support toolchain files with scikit-build-core yet (meaning they aren't really tested, they do get used for Emscripten, so I think they work).

Footnotes

1. Standard here just means we follow what other backends and builders do. That's standard on macOS since Apple defines things like MACOSX_DEPLOYMENT_TARGET. It's much messier for Windows and linux, though. And it's non-trivial for something like Windows where you also have to tell the backend exactly which Python library to link to when cross compiling! There are efforts to make a standard for cross-compiling Python packages. If that happens, supporting that will be our priority. ↩

[michaeltryby]

Thanks for the clarification. I appreciate your efforts. Keep up the good work!

[LecrisUT]

Just to emphasise, we are not dependent on cibuildwheel workflow and we want to support as many workflows as possible and try to work with the developers there as closely as possible. If you are developing a tool that builds/distributes a python project do let us know with a link to the project and what best practices and limitations you have.

I for example am involved in the Fedora packaging ecosystem and try to make that process as easy as possible. One workflow that we are not covering yet, but we should, is yocto project (somebody is working on it I think). But other than that I don't think we have any other projects on our radar, so do let us know if you think we should check it.

[henryiii]

Update: we need to know the wheel tag before we make a build directory, since we allow build/<wheel_tag> as a possible build directory, so asking CMake is out (was thinking we might be able to use the file_api for this!). We'd have to process every possible way to set this manually. I don't think it's a good idea until we at least have first-class support for toolchains and presets.