Add support for db fixture (test inside transaction) for asyncio tests #1223

lode-braced · 2025-08-12T08:41:36Z

In using and debugging pytest-django with async tests & fixtures, I ended up hacking together a way to get the db fixture working by enabling the transaction in the sync to async executor thread in which async orm queries run.

This PR aims to integrate that hack as actual functionality into pytest-django.

The async work is done, I also worked to refine the sync side database access: not allowing db access to threads other than the main thread which is running the test (& transaction). That second part is causing some test failures I've yet been unable to fix on 3.9 & django 4.2, will try to fix or drop it from the PR.

edit: I managed to fix the issue with the test failures: my monkeypatches were dropping the self attribute. Ready for review.

Suggestions & feedback welcome.

…n using TestCase (i.e., not transactional)

This reverts commit 3ee56c4.

This reverts commit 3f1fe98.

…c & async fixtures

This reverts commit 7673814.

This reverts commit cca2dff.

…functionality for django<5.1

kingbuzzman

This is not a code review

From what I can briefly see; this looks great, i've personally had this issue come up, so happy you've taken this on!

lode-braced · 2025-08-12T10:36:41Z

@kingbuzzman CI and myself are now happy with where the branch is at, feel free to have a deeper look now.

kingbuzzman · 2025-08-12T15:12:26Z

@lode-braced can you please sync with main, there are new CI rules

…c_fixture

lode-braced · 2025-08-12T15:50:40Z

@lode-braced can you please sync with main, there are new CI rules

Done!

tox.ini

kingbuzzman

5/8 files reviewed. More to follow.

kingbuzzman · 2025-08-14T08:56:55Z

pyproject.toml

@@ -53,6 +53,10 @@ coverage = [
    "coverage[toml]",
    "coverage-enable-subprocess",
 ]
+async = [
+    "asgiref>=3.9.1",


This is a django requirement, not ours.

Suggested change

"asgiref>=3.9.1",

I wrote this before i saw, you use it in fixtures.py now im in two minds... I don't have a strong opinion. I'll leave this up for discussion.

kingbuzzman · 2025-08-14T08:58:26Z

pytest_django/fixtures.py

@@ -3,7 +3,7 @@
 from __future__ import annotations

 import os
-from collections.abc import Generator, Iterable, Sequence
+from collections.abc import AsyncGenerator, Generator, Iterable, Sequence


Please move AsyncGenerator inside if TYPE_CHECKING: -- i will be updating the linter at some point...

Suggested change

from collections.abc import AsyncGenerator, Generator, Iterable, Sequence

from collections.abc import Generator, Iterable, Sequence

kingbuzzman · 2025-08-14T09:07:07Z

tests/test_async_db.py

+@pytest.mark.parametrize("run_number", [1, 2])
+@pytestmark
+@pytest.mark.django_db
+async def test_async_db(db, run_number) -> None:


Not needed

Suggested change

async def test_async_db(db, run_number) -> None:

async def test_async_db(run_number) -> None:

kingbuzzman · 2025-08-14T09:07:19Z

tests/test_async_db.py

+
+
+@fixturemark
+async def db_item(db) -> Any:


Not needed

Suggested change

async def db_item(db) -> Any:

async def db_item() -> Any:

kingbuzzman · 2025-08-14T09:07:28Z

tests/test_async_db.py

+
+
+@pytest.fixture
+def sync_db_item(db) -> Any:


Not needed

Suggested change

def sync_db_item(db) -> Any:

def sync_db_item() -> Any:

kingbuzzman · 2025-08-14T09:18:17Z

tests/test_async_db.py

+
+@pytestmark
+@pytest.mark.xfail(strict=True, reason="Sync fixture used in async test")
+async def test_db_item(db_item: Item, sync_db_item) -> None:


Suggested change

async def test_db_item(db_item: Item, sync_db_item) -> None:

@pytest.mark.usefixtures("db_item", "sync_db_item")

async def test_db_item() -> None:

kingbuzzman · 2025-08-14T09:18:31Z

tests/test_async_db.py

+
+
+@pytest.mark.xfail(strict=True, reason="Async fixture used in sync test")
+def test_sync_db_item(async_db_item: Item, sync_db_item) -> None:


Suggested change

def test_sync_db_item(async_db_item: Item, sync_db_item) -> None:

@pytest.mark.usefixtures("db_item", "sync_db_item")

def test_sync_db_item() -> None:

kingbuzzman · 2025-08-14T09:45:14Z

pytest_django/plugin.py

+    def _unblocked_sync_only(self, wrapper_self: Any, *args, **kwargs):
+        __tracebackhide__ = True
+        if threading.current_thread() != threading.main_thread():
+            raise RuntimeError(


I don't necessarily think this is bad, but this is definitely new functionality that would be better in another PR that can be studied independently. @bluetech thoughts?

kingbuzzman · 2025-08-14T09:46:23Z

tests/test_db_thread_safeguards.py

+    def worker() -> None:
+        try:
+            # Any ORM operation that touches the DB will attempt to ensure a connection.
+            # This should raise from the "sync_only" db blocker in non-main threads.
+            Item.objects.count()
+        except BaseException as exc:  # noqa: BLE001 - we want to capture exactly what is raised
+            captured[0] = exc
+
+    t = threading.Thread(target=worker)
+    t.start()
+    t.join()
+
+    assert captured[0] is not None, "Expected DB access in worker thread to raise an exception"
+    assert isinstance(captured[0], RuntimeError)
+    assert "only allowed in the main thread" in str(captured[0])


[opinion] This is.. a lot. I think something like this would be clearer.

from concurrent.futures import ThreadPoolExecutor @pytest.mark.django_db def test_sync_db_access_in_non_main_thread_is_blocked() -> None: """Ensure that sync DB access from a non-main thread is not allowed.""" Item.objects.create(name="spam") with ThreadPoolExecutor() as executor: future = executor.submit(Item.objects.count) with pytest.raises(RuntimeError, match="only allowed in the main thread"): future.result() @pytest.mark.django_db(transaction=True) def test_transactional_db_access_in_non_main_thread_is_allowed() -> None: """Ensure that transactional DB access from a non-main thread is allowed.""" Item.objects.create(name="spam") with ThreadPoolExecutor() as executor: future = executor.submit(Item.objects.count) assert future.result() == 1

PS this should be in the other PR I mentioned above.

Lode Rosseel added 9 commits August 11, 2025 16:54

Add async support: run transaction start/stop in an async context whe…

58f7f24

…n using TestCase (i.e., not transactional)

Revertme: test pipeline

3ee56c4

Move paramspec into type check block for python 3.9

3515460

Revertme: remove cov step

3f1fe98

Fix type hints

1a09850

Add docs for async, thread checks, refer to docs in plugin errors

629cafd

Revert "Revertme: test pipeline"

4a79664

This reverts commit 3ee56c4.

Revert "Revertme: remove cov step"

f46bac0

This reverts commit 3f1fe98.

Drop sync extra checks

7673814

lode-braced marked this pull request as draft August 12, 2025 08:48

Lode Rosseel added 7 commits August 12, 2025 10:55

Fix docs layout

061ca4b

Remove unblock kwarg from sync

cca2dff

Reduce code duplication: reuse test case class creation logic for syn…

1bf9be6

…c & async fixtures

Revert "Drop sync extra checks"

0525d62

This reverts commit 7673814.

Revert "Remove unblock kwarg from sync"

2c1d21b

This reverts commit cca2dff.

Wrap patched method with one containing explicit wrapped self to fix …

95b03a0

…functionality for django<5.1

Chore: reformat

b219fcb

kingbuzzman reviewed Aug 12, 2025

View reviewed changes

Lode Rosseel added 4 commits August 12, 2025 12:11

Chore: update type hints

593d02f

Add test case for db access outside main thread in sync context

3c986bf

Add unblock test case when both flags are set

0fb5f38

chore: format

4b1274c

lode-braced marked this pull request as ready for review August 12, 2025 10:26

lode-braced mentioned this pull request Aug 12, 2025

Database transactions with asyncio #580

Open

lode-braced requested a review from kingbuzzman August 12, 2025 10:35

Merge remote-tracking branch 'refs/remotes/origin/main' into fix/asyn…

215c679

…c_fixture

kingbuzzman reviewed Aug 12, 2025

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

Lode Rosseel added 2 commits August 12, 2025 21:32

Merge remote-tracking branch 'origin/main' into fix/async_fixture

cff3a16

Chore: fix tox indenting

9aff5fd

kingbuzzman reviewed Aug 14, 2025

View reviewed changes

	from collections.abc import AsyncGenerator, Generator, Iterable, Sequence
	from collections.abc import Generator, Iterable, Sequence

	async def test_async_db(db, run_number) -> None:
	async def test_async_db(run_number) -> None:

	async def test_db_item(db_item: Item, sync_db_item) -> None:
	@pytest.mark.usefixtures("db_item", "sync_db_item")
	async def test_db_item() -> None:



		@pytest.mark.xfail(strict=True, reason="Async fixture used in sync test")
		def test_sync_db_item(async_db_item: Item, sync_db_item) -> None:

	def test_sync_db_item(async_db_item: Item, sync_db_item) -> None:
	@pytest.mark.usefixtures("db_item", "sync_db_item")
	def test_sync_db_item() -> None:

Add support for db fixture (test inside transaction) for asyncio tests #1223

Are you sure you want to change the base?

Add support for db fixture (test inside transaction) for asyncio tests #1223

Conversation

lode-braced commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kingbuzzman left a comment

Choose a reason for hiding this comment

Uh oh!

lode-braced commented Aug 12, 2025

Uh oh!

kingbuzzman commented Aug 12, 2025

Uh oh!

lode-braced commented Aug 12, 2025

Uh oh!

Uh oh!

kingbuzzman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lode-braced commented Aug 12, 2025 •

edited

Loading