Support for configuring CMake arguments in pip install without environment variables

[jameslamb]

Short Question

Is it possible to pass through command-line options from pip install to CMake via scikit-build without using environment variables?

---

More Details

I'm working on migrating a CMake-based project that uses setuptools + setup.py + subclassing of stuff from stuff from setuptools.command to a PEP517/PEP518 compliant build.

Because the project relies on CMake to build its C/C++ shared library, I'd like to use scikit-build for this.

Today, the project supports passing custom arguments through pip install to change the library compiled when building from an sdist.

For example,

pip install --install-option=--use-gpu

becomes a command like the following executed by a subprocess.check_call() call in setup.py

cmake -DUSE_GPU=ON ..

I'd like to achieve similar behavior using scikit-build.

I've seen https://scikit-build.readthedocs.io/en/latest/usage.html#cmake-configure-options, which suggests that one can set a CMAKE_ARGS environment variable.

I'd like to avoid recommending that to users of my project, because:

1. that variable name applies to all CMake builds, not just my project or just scikit-build projects
   • as mentioned in Respecting CMAKE_ARGS environment variable is unexpected #722
   • reference in CMake docs: https://cmake.org/cmake/help/v3.3/module/ExternalProject.html
2. I've seen firsthand that some users of my project don't understand what environment variables are or how to set them for a process

Specific project I'm working on:

• setup.py: https://github.com/microsoft/LightGBM/blob/87e3c0d9ba81ee229dedeaac10765054c25b5284/python-package/setup.py
• discussion about the future of how the project builds wheels: [RFC] [python] 'setup.py install' is deprecated lightgbm-org/LightGBM#5061
• draft PR showing my efforts towards this: WIP: [python-package] switch to scikit-build and pyproject.toml (fixes #5061) lightgbm-org/LightGBM#5623

---

Notes for Reviewers

I hope I'm in the right place (assumed based on #800 that this Discussions board is the preferred place to post questions like this).

Thanks for your time and considerations, and for this awesome project!

[henryiii]

This is currently solved in scikit-build-core via config-settings options:

pip install --config-settings=cmake.define.USE_GPU=ON

There's no matching system for the setuptools plugin yet; developing that will likely affect what scikit-build classic will adopt (since it will move to using that). Setuptools has the annoying feature of requiring an extra layer on config settings options (global-option and such).

--install-options is going away (actually, I think the last version or two of pip already doesn't support it? If not, then the one that switches to PEP 517 by default will likely be the one to drop it?)

And to point 1): CMAKE_ARGS does not apply to all CMake builds. It's not a special CMake variable. It's pretty commonly used for CMake args, but it's not a special environment variable for CMake. (It's a keyword in ExternalProject/FetchContent, but not sure how that relates to environment variables).

[jameslamb]

Thank you so much! Apologies, I didn't understand the relationship between scikit-build-core and scikit-build, so didn't think to look for information about this over in https://github.com/scikit-build/scikit-build-core. I understaand now that:

• this is documented there, in https://github.com/scikit-build/scikit-build-core#configuration
• scikit-build-core can be used directly as a build backend
  • I'd thought incorrectly that it was just an implementation detail of scikit-build, not intended to be used standalone

The --config-settings support in scikit-build-core, looks like exactly what I want!

CMAKE_ARGS does not apply to all CMake builds. ... It's a keyword in ExternalProject/FetchContent, but not sure how that relates to environment variables

Fair, I was definitely misinterpreting that doc about ExternalProject. I often get confused about what CMake will read from the environment and what are truly CMake variables that you need to, for example, assign with set() in a CMake script. Thanks for correcting my understanding!

I do still think it's at least true that environment variable CMAKE_ARGS will affect all scikit-build builds. That probably isn't likely to cause issues in the environment where you're building a wheel for one project to be published to a repository like PyPI, but could be a source of surprises for a situation where someone runs something like the following

export CMAKE_ARGS="..."
pip install pkgA pkgB

on a platform where wheels are not available for pkgA or `pkgB.

In that situation, both the source distributions for pkgA and pkgB are going to both be downloaded and wheels will be built locally from them. If they both use scikit-build, then wheel-building for both will be affected by the value of the CMAKE_ARGS environment variable.