-p no:randomly causing tests to fail under Python 3.8 and 3.9

[alexandrul]

Python Version

3.9.20

pytest Version

8.3.2

Package Version

3.15.0

Description

As you can see in the following table, having pytest-randomly installed and using -p no:randomly results in failed tests but only under Python 3.8 and 3.9:

	Python 3.8 / 3.9	Python 3.10 / 3.11 / 3.12
pytest -p no:randomly tests/test_jsonreport.py::test_report_collectors	fail	pass
pytest tests/test_jsonreport.py::test_report_collectors	pass	pass

To replicate the issue, checkout this repo and run the above commands in a Python 3.9 virtualenv.

[alexandrul]

Output for passing tests:

❯ pytest tests/test_jsonreport.py::test_report_collectors
======================================== test session starts =========================================
platform linux -- Python 3.9.20, pytest-8.3.2, pluggy-1.5.0
Using --randomly-seed=2524090775
rootdir: ~/pytest-json-report-wip-3.9/repo
plugins: metadata-3.1.1, anyio-4.4.0, randomly-3.15.0, xdist-3.6.1, flaky-3.8.1, json-report-wip-1.5.1
collected 3 items                                                                                    

tests/test_jsonreport.py ...                                                                   [100%]

========================================= 3 passed in 0.87s ==========================================

Output for failing tests:

❯ pytest -p no:randomly tests/test_jsonreport.py::test_report_collectors
======================================== test session starts =========================================
platform linux -- Python 3.9.20, pytest-8.3.2, pluggy-1.5.0
rootdir: ~/pytest-json-report-wip-3.9/repo
plugins: metadata-3.1.1, anyio-4.4.0, xdist-3.6.1, flaky-3.8.1, json-report-wip-1.5.1
collected 3 items                                                                                    

tests/test_jsonreport.py .FF                                                                   [100%]

============================================== FAILURES ==============================================
...
==================================== 2 failed, 1 passed in 0.24s =====================================

[adamchainz]

This sounds to me like you have some flaky tests? I don't have time to work on your project. If you can produce a minimal reproduction that proves it's an issue in pytest-randomly, sure, I can look at it.

[alexandrul]

• I don't want you to work on my project
• however, I would like to migrate to hatch and use it for tests too, instead of tox, but hatch uses your project
• I couldn't find a simpler case to reproduce the issue
• everything works just fine if randomly is not installed in the virtualenv
• if randomly is installed and I'm not using -p no:randomly then the tests are passing under all supported Python versions
• if randomly is installed and I'm using -p no:randomly then the tests are failing under Python 3.8/3.9 and are passing for Python 3.10+

I couldn't find out the root cause yet but I've seen this conditional import and it's an interesting coincidence: https://github.com/pytest-dev/pytest-randomly/blob/main/src/pytest_randomly/__init__.py#L22-L25

[adamchainz]

If you can provide a minimal reproducer, I can look at that. But I don't have time to clone your whole project and try debug inside unfamiliar tools like Hatch.