indirectly parameterized session-scoped fixtures aren't

[jclerman]

Summary

When a fixture is passed parameters from a test, via @pytest.mark.parameterize's indirect=True, per-parameter-value session-scoping does not work. For a given passed value, the fixture may be created/torn down multiple times.

Minimal example

Code

import pytest
from _pytest.fixtures import FixtureRequest

@pytest.fixture(scope="session")
def case_fixture(request: FixtureRequest):
    n = request.param
    print(f"Setting up case {n}")
    yield f"case_fixture {n}"
    print(f"Tearing down case {n}")

@pytest.mark.parametrize(
    "case_fixture, expected",
    [
        (1, "case_fixture 1"),
        (2, "case_fixture 2"),
        (1, "case_fixture 1"),
    ],
    indirect=["case_fixture"],
    scope="session"
)
def test_one(case_fixture, expected):
    assert case_fixture == expected

Expected Result

% pytest -k test_fixture_p -s
Test session starts (platform: darwin, Python 3.11.10, pytest 8.2.2, pytest-sugar 1.0.0)
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/jclerman/git/backend-integration-tester
configfile: pyproject.toml
plugins: ddtrace-2.14.4, cov-5.0.0, sugar-1.0.0, icdiff-0.9, benchmark-4.0.0, anyio-4.6.2.post1, asyncio-0.25.1
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function
collected 3 items / 0 deselected / 3 selected
Setting up case 1
 tests/test_fixture_parameterization.py ✓
Setting up case 2
 tests/test_fixture_parameterization.py ✓✓
Tearing down case 2
Tearing down case 1
 tests/test_fixture_parameterization.py ✓✓✓

Actual Result

% pytest -k test_fixture_p -s
Test session starts (platform: darwin, Python 3.11.10, pytest 8.2.2, pytest-sugar 1.0.0)
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/jclerman/git/backend-integration-tester
configfile: pyproject.toml
plugins: ddtrace-2.14.4, cov-5.0.0, sugar-1.0.0, icdiff-0.9, benchmark-4.0.0, anyio-4.6.2.post1, asyncio-0.25.1
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function
collected 3 items / 0 deselected / 3 selected
Setting up case 1
 tests/test_fixture_parameterization.py ✓
Tearing down case 1
Setting up case 2
 tests/test_fixture_parameterization.py ✓✓
Tearing down case 2
Setting up case 1
Tearing down case 1
 tests/test_fixture_parameterization.py ✓✓✓

Platform details

Pytest 8.2.2
Python 3.11.10
MacOS 15.2 (Sequoia)

Pip list output

% pip list
Package                       Version     Editable project location
----------------------------- ----------- ----------------------------------------------
alabaster                     0.7.16
annotated-types               0.7.0
anyio                         4.6.2.post1
arq                           0.26.1
Babel                         2.15.0
backend_integration_tester    0.1.0       /Users/jclerman/git/backend-integration-tester
bytecode                      0.15.1
cachetools                    5.5.0
certifi                       2024.7.4
cfgv                          3.4.0
chardet                       5.2.0
charset-normalizer            3.3.2
click                         8.1.7
colorama                      0.4.6
coverage                      7.6.0
ddtrace                       2.14.4
Deprecated                    1.2.14
distlib                       0.3.8
dnspython                     2.7.0
docutils                      0.18.1
email_validator               2.2.0
envier                        0.6.1
fastapi                       0.115.3
filelock                      3.15.4
h11                           0.14.0
hiredis                       3.1.0
httpcore                      1.0.6
httpx                         0.27.2
icdiff                        2.0.7
identify                      2.6.0
idna                          3.7
imagesize                     1.4.1
importlib_metadata            8.4.0
iniconfig                     2.0.0
Jinja2                        3.1.4
MarkupSafe                    2.1.5
mypy                          1.14.1
mypy-extensions               1.0.0
nest-asyncio                  1.6.0
nodeenv                       1.9.1
opentelemetry-api             1.27.0
packaging                     24.1
pip                           24.3.1
platformdirs                  4.2.2
pluggy                        1.5.0
pprintpp                      0.4.0
pre-commit                    3.7.1
protobuf                      5.28.3
py-cpuinfo                    9.0.0
pydantic                      2.10.4
pydantic_core                 2.27.2
pydantic-extra-types          2.10.2
Pygments                      2.18.0
pyproject-api                 1.8.0
pytest                        8.2.2
pytest-asyncio                0.25.1
pytest-benchmark              4.0.0
pytest-cov                    5.0.0
pytest-icdiff                 0.9
pytest-sugar                  1.0.0
python-dateutil               2.9.0.post0
PyYAML                        6.0.1
re-assert                     1.1.0
redis                         5.2.1
regex                         2024.11.6
requests                      2.32.3
sentry-sdk                    2.17.0
setuptools                    75.6.0
sironabackend                 0.66.0
six                           1.17.0
sniffio                       1.3.1
snowballstemmer               2.2.0
Sphinx                        7.3.7
sphinx-rtd-theme              1.3.0
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
starlette                     0.41.0
termcolor                     2.4.0
toml                          0.10.2
tox                           4.23.2
typing_extensions             4.12.2
tzdata                        2024.2
urllib3                       2.2.2
uvicorn                       0.32.0
uvloop                        0.21.0
virtualenv                    20.26.3
wrapt                         1.16.0
xmltodict                     0.14.2
zipp                          3.20.2

[RonnyPfannschmidt]

Pytest cannot know if a duplicate is intentional or accidentally

See pytest -v

Also see --setup-plan

[RonnyPfannschmidt]

A important note I forgot

Multiple parameter variants of a fixture may not exist at the same time

[jclerman]

A important note I forgot

Multiple parameter variants of a fixture may not exist at the same time

I think this latter point is the limit I'm running into. If that could be overcome, it would be very helpful for my use cases. Doing so would sidestep any issues related to case (or test) ordering.

Would it be possible to fix that, either in pytest or in a plugin?

[RonnyPfannschmidt]

There's currently no plan/concepts for such fixtures in a builtin way

It's so deep in pytest that a plugin can't safely mess with it

A workaround could be a managing fixture that creates data and essentially having the concrete parameterized fixture leak so the managing fixture has to clean up all instances

[github-actions]

This issue is stale because it has the status: needs information label and requested follow-up information was not provided for 14 days.

[Anton3]

As @RonnyPfannschmidt said, the current invariant is that no more than 1 instance of each fixture exists at a time. (I hope pytest does not change this invariant, as some of our parametrized fixtures cannot have multiple concurrent instances, they would conflict with each other, e.g. as files on disk.)

So pytest walks through the tests, in whatever order they end up (this can be changed via pytest_collection_modifyitems) and sets up / tears down fixtures, given the invariant above.

The issue can be mitigated by reordering test parameters so that for "heavy" parameters, same values are grouped together:

@pytest.mark.parametrize(
    "case_fixture, expected",
    [
        (1, "case_fixture 1"),
        (1, "case_fixture 1"),  # <====
        (2, "case_fixture 2"),  # <====
    ],
    indirect=["case_fixture"],
    scope="session"
)
def test_one(case_fixture, expected):
    assert case_fixture == expected

This can be automated using pytest_collection_modifyitems:

from collections import defaultdict
from typing import List

def pytest_collection_modifyitems(items: List[pytest.Item]):
    grouped = defaultdict(list)
    for item in items:
        grouping_key = None
        if hasattr(item, 'callspec'):
            # callspec does not seem to be public API, but oh well
            grouping_key = item.callspec.params.get('my_heavy_fixture', None)
        grouped[grouping_key].append(item)
    items.clear()
    items.extend(item for group in grouped.values() for item in group)

If there are multiple "heavy" fixtures with conflicting order of values, then there is nothing that can be done.