Locking different versions of a dependency for different python versions

[aryarm]

I use nox-poetry to install the locked versions of our dependencies in our CI, which tests against python 3.7+. This has been working great, except that I haven't been able to lock my dependencies for all versions of python -- just some of them. The issue stems from the fact that some of the dependencies in my lock file only support a subset of the python versions that I'd like to test.

How I usually lock my dependencies

For example, let's say that my pyproject.toml looks like this:

requires-python = ">=3.7.1"
dependencies = [
    "numpy>=1.17.3",
]

If I want to upgrade numpy in my lock file to 1.21.1, I can usually just do this:

1. Temporarily pin numpy to 1.21.1 in my pyproject.toml file

   dependencies = [
       "numpy==1.21.1",
   ]
   

2. Run poetry lock
3. Revert the pyproject.toml file to a min version requirement of 1.17.3

   dependencies = [
       "numpy>=1.17.3",
   ]
   

4. Run poetry lock again

Now I have 1.21.1 in my lock file and 1.17.3 in my pyproject.toml file. That's great.

But I prefer to keep my project locked to the minimum versions of all of my dependencies at all times. This allows me to test in our CI that changes to our code won't require us to bump the minimum versions of our dependencies. So, at the moment, my pyproject.toml file has a minimum version specifier for numpy 1.17.3 and my poetry.lock file contains numpy 1.17.3.

My current issue

Now, let's say that I'd like to test my project against python 3.7 and 3.9. Unfortunately, numpy 1.17.3 doesn't support python 3.9, but numpy 1.21.1 does. Of course, I can lock numpy at 1.21.1, but I would prefer to do that only for python 3.9 and not 3.7 and 3.8. I'm able to achieve this temporarily by using python markers. Here's what I'm doing, which mirrors the steps above.

1. Temporarily pin numpy to 1.21.1 for py3.9 and 1.17.3 for py3.7-3.8 in my pyproject.toml file

   dependencies = [
       "numpy==1.17.3 ; python_version < '3.9'",
       "numpy==1.21.1 ; python_version >= '3.9'",
   ]
   

2. Run poetry lock. Now, both versions of numpy are in my lock file, as I would like.
3. Revert the pyproject.toml file to a min version requirement of 1.17.3

   dependencies = [
       "numpy>=1.17.3",
   ]
   

4. Run poetry lock again

But this doesn't work! After step 4, only one version of numpy remains in my lock file, not both :(

I also tried the following for step 3 instead:

dependencies = [
   "numpy>=1.17.3 ; python_version < '3.9'",
   "numpy>=1.21.1 ; python_version >= '3.9'",
]

But this gives me the same result after step 4.

Do you have any ideas for how I can achieve what I want? Or is it just not possible?

[radoering]

You can use both, project.dependencies and tool.poetry.dependencies, see https://python-poetry.org/docs/dependency-specification/#projectdependencies-and-toolpoetrydependencies
When using both, tool.poetry.dependencies do not influence the metadata but are only used for locking.

I would try something like:

[project]
dependencies = [
    "numpy>=1.17.3",
]

[tool.poetry.dependencies]
numpy = [
    { version = "1.17.3", markers = "python_version < '3.9'" },
    { version = "1.21.1", markers = "python_version >= '3.9'" },
]

[aryarm]

Thank you so much! This was the solution I've been looking for this entire time.