BLD Specify build time dependencies via pyproject.toml (scikit-learn#16244)

* add pyproject.toml

* do it in CI

* cln

* no build isolation in the doc

* bump dependencies

* no-build-isolation everywhere needed

Author: jeremiedbb

MANIFEST.in

@@ -5,3 +5,4 @@ recursive-include sklearn *.c *.h *.pyx *.pxd *.pxi *.tp
 recursive-include sklearn/datasets *.csv *.csv.gz *.rst *.jpg *.txt *.arff.gz *.json.gz
 include COPYING
 include README.rst
+include pyproject.toml

build_tools/azure/install.sh

@@ -90,7 +90,6 @@ elif [[ "$DISTRIB" == "conda-pip-latest" ]]; then
     # conda is still used as a convenient way to install Python and pip.
     make_conda "python=$PYTHON_VERSION"
     python -m pip install -U pip
-    python -m pip install numpy scipy cython joblib
     python -m pip install pytest==$PYTEST_VERSION pytest-cov pytest-xdist
     python -m pip install pandas matplotlib pyamg
     # do not install dependencies for lightgbm since it requires scikit-learn
@@ -121,7 +120,15 @@ except ImportError:
 "
 python -m pip list
 
-# Use setup.py instead of `pip install -e .` to be able to pass the -j flag
-# to speed-up the building multicore CI machines.
-python setup.py build_ext --inplace -j 3
-python setup.py develop
+if [[ "$DISTRIB" == "conda-pip-latest" ]]; then
+    # Check that pip can automatically install the build dependencies from
+    # pyproject.toml using an isolated build environment:
+    pip install --verbose --editable .
+else
+    # Use the pre-installed build dependencies and build directly in the
+    # current environment.
+    # Use setup.py instead of `pip install -e .` to be able to pass the -j flag
+    # to speed-up the building multicore CI machines.
+    python setup.py build_ext --inplace -j 3
+    python setup.py develop
+fi

build_tools/circle/build_test_pypy.sh

@@ -34,7 +34,7 @@ export LOKY_MAX_CPU_COUNT="2"
 export OMP_NUM_THREADS="1"
 
 python setup.py build_ext --inplace -j 3
-pip install -e .
+pip install --no-build-isolation -e .
 
 # Check that Python implementation is PyPy
 python - << EOL

doc/developers/advanced_installation.rst

@@ -59,7 +59,7 @@ feature, code or documentation improvement).
 #. Install Cython_ and build the project with pip in :ref:`editable_mode`::
 
         pip install cython
-        pip install --verbose --editable .
+        pip install --verbose --no-build-isolation --editable .
 
 #. Check that the installed scikit-learn has a version number ending with
    `.dev0`::
@@ -71,8 +71,11 @@ feature, code or documentation improvement).
 
 .. note::
 
-    You will have to re-run the ``pip install --editable .`` command every time
-    the source code of a Cython file is updated (ending in `.pyx` or `.pxd`).
+    You will have to run the ``pip install --no-build-isolation --editable .``
+    command every time the source code of a Cython file is updated
+    (ending in `.pyx` or `.pxd`). Use the ``--no-build-isolation`` flag to
+    avoid compiling the whole project each time, only the files you have
+    modified.
 
 Dependencies
 ------------
@@ -152,9 +155,9 @@ Editable mode
 
 If you run the development version, it is cumbersome to reinstall the package
 each time you update the sources. Therefore it is recommended that you install
-in with the ``pip install --editable .`` command, which allows you to edit the
-code in-place. This builds the extension in place and creates a link to the
-development directory (see `the pip docs
+in with the ``pip install --no-build-isolation --editable .`` command, which
+allows you to edit the code in-place. This builds the extension in place and
+creates a link to the development directory (see `the pip docs
 <https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs>`_).
 
 This is fundamentally similar to using the command ``python setup.py develop``
@@ -207,7 +210,7 @@ environment variables in the current command prompt.
 
 Finally, build scikit-learn from this command prompt::
 
-    pip install --verbose --editable .
+    pip install --verbose --no-build-isolation --editable .
 
 .. _compiler_macos:
 
@@ -240,7 +243,7 @@ scikit-learn from source::
         "conda-forge::compilers>=1.0.4" conda-forge::llvm-openmp
     conda activate sklearn-dev
     make clean
-    pip install --verbose --editable .
+    pip install --verbose --no-build-isolation --editable .
 
 .. note::
 
@@ -302,7 +305,7 @@ Finally, build scikit-learn in verbose mode (to check for the presence of the
 ``-fopenmp`` flag in the compiler commands)::
 
     make clean
-    pip install --verbose --editable .
+    pip install --verbose --no-build-isolation --editable .
 
 .. _compiler_linux:
 
@@ -351,7 +354,7 @@ in the user folder using conda::
 
     conda create -n sklearn-dev numpy scipy joblib cython conda-forge::compilers
     conda activate sklearn-dev
-    pip install --verbose --editable .
+    pip install --verbose --no-build-isolation --editable .
 
 .. _compiler_freebsd:
 
@@ -374,7 +377,7 @@ can set the environment variables to these locations::
 
 Finally, build the package using the standard command::
 
-    pip install --verbose --editable .
+    pip install --verbose --no-build-isolation --editable .
 
 For the upcoming FreeBSD 12.1 and 11.3 versions, OpenMP will be included in
 the base system and these steps will not be necessary.

doc/developers/contributing.rst

@@ -219,7 +219,7 @@ how to set up your git repository:
 
 5. Install scikit-learn in editable mode::
 
-       $ pip install --editable .
+       $ pip install --no-build-isolation --editable .
 
    for more details about advanced installation, see the
    :ref:`install_bleeding_edge` section.
@@ -266,8 +266,13 @@ modifying code and submitting a PR:
 
 .. note::
 
-  If you are modifying a Cython module, you have to re-run step 5 after modifications
-  and before testing them.
+    If you are modifying a Cython module, you have to re-compile after
+    modifications and before testing them::
+
+        pip install --no-build-isolation -e .
+
+    Use the ``--no-build-isolation`` flag to avoid compiling the whole project
+    each time, only the files you have modified.
 
 It is often helpful to keep your local feature branch synchronized with the
 latest changes of the main scikit-learn repository::

pyproject.toml

@@ -0,0 +1,9 @@
+[build-system]
+# Minimum requirements for the build system to execute.
+requires = [
+    "setuptools",
+    "wheel",
+    "Cython>=0.28.5",
+    "numpy>=1.13.3",
+    "scipy>=0.19.1",
+]