Skip to content

Commit c7f5779

Browse files
SalmonModeSalmonMode
committed
Allow defining custom logic for test distribution among groups and reordering test groups for execution
1 parent 6fd5b56 commit c7f5779

File tree

8 files changed

+798
-589
lines changed

8 files changed

+798
-589
lines changed

Diff for: ‎README.rst

+124-11
Original file line numberDiff line numberDiff line change
@@ -80,20 +80,133 @@ that worker and report the failure as usual. You can use the
8080
``--max-worker-restart`` option to limit the number of workers that can
8181
be restarted, or disable restarting altogether using ``--max-worker-restart=0``.
8282

83-
By default, the ``-n`` option will send pending tests to any worker that is available, without
84-
any guaranteed order, but you can control this with these options:
83+
Dividing tests up
84+
^^^^^^^^^^^^^^^^^
85+
86+
In order to divide the tests up amongst the workers, ``pytest-xdist`` first puts sets of
87+
them into "test groups". The tests within a test group are all run together in one shot,
88+
so fixtures of larger scopes won't be run once for every single test. Instead, they'll
89+
be run as many times as they need to for the tests within that test group. But, once
90+
that test group is finished, it should be assumed that all cached fixture values from
91+
that test group's execution are destroyed.
92+
93+
By default, there is no grouping logic and every individual test is placed in its own
94+
test group, so using the ``-n`` option will send pending tests to any worker that is
95+
available, without any guaranteed order. It should be assumed that when using this
96+
approach, every single test is run entirely in isolation from the others, meaning the
97+
tests can't rely on cached fixture values from larger-scoped fixtures.
98+
99+
Provided test grouping options
100+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
101+
102+
By default, ``pytest-xdist`` doesn't group any tests together, but it provides some
103+
grouping options, based on simple criteria about a test's nodeid. so you can gunarantee
104+
that certain tests are run in the same process. When they're run in the same process,
105+
you gunarantee that larger-scoped fixtures are only executed as many times as would
106+
normally be expected for the tests in the test group. But, once that test group is
107+
finished, it should be assumed that all cached fixture values from that test group's
108+
execution are destroyed.
109+
110+
Here's the options that are built in:
111+
112+
* ``--dist=loadscope``: tests will be grouped by **module** shown in each test's node
113+
for *test functions* and by the **class** shown in each test's nodeid for *test
114+
methods*. This feature was added in version ``1.19``.
115+
116+
* ``--dist=loadfile``: tests will be grouped by the **module** shown in each test's
117+
nodeid. This feature was added in version ``1.21``.
118+
119+
Defining custom load distribution logic
120+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
121+
122+
``pytest-xdist`` iterates over the entire list of collected tests and usually determines
123+
what group to put them in based off of their nodeid. There is no set number of test
124+
groups, as it creates a new groups as needed. You can tap into this system to define
125+
your own grouping logic by using the ``pytest_xdist_set_test_group_from_nodeid``.
126+
127+
If you define your own copy of that hook, it will be called once for every test, and the
128+
nodeid for each test will be passed in. Whatever it returns is the test group for that
129+
test. If a test group doesn't already exist with that name, then it will be created, so
130+
anything can be used.
131+
132+
For example, let's say you have the following tests::
133+
134+
test/test_something.py::test_form_upload[image-chrome]
135+
test/test_something.py::test_form_upload[image-firefox]
136+
test/test_something.py::test_form_upload[video-chrome]
137+
test/test_something.py::test_form_upload[video-firefox]
138+
test/test_something_else.py::test_form_upload[image-chrome]
139+
test/test_something_else.py::test_form_upload[image-firefox]
140+
test/test_something_else.py::test_form_upload[video-chrome]
141+
test/test_something_else.py::test_form_upload[video-firefox]
142+
143+
In order to have the ``chrome`` related tests run together and the ``firefox`` tests run
144+
together, but allow them to be separated by file, this could be done:
85145

86-
* ``--dist=loadscope``: tests will be grouped by **module** for *test functions* and
87-
by **class** for *test methods*, then each group will be sent to an available worker,
88-
guaranteeing that all tests in a group run in the same process. This can be useful if you have
89-
expensive module-level or class-level fixtures. Currently the groupings can't be customized,
90-
with grouping by class takes priority over grouping by module.
91-
This feature was added in version ``1.19``.
146+
.. code-block:: python
147+
148+
def pytest_xdist_set_test_group_from_nodeid(nodeid):
149+
browser_names = ['chrome', 'firefox']
150+
nodeid_params = nodeid.split('[', 1)[-1].rstrip(']').split('-')
151+
for name in browser_names:
152+
if name in nodeid_params:
153+
return "{test_file}[{browser_name}]".format(
154+
test_file=nodeid.split("::", 1)[0],
155+
browser_name=name,
156+
)
157+
158+
The tests would then be divided into these test groups:
159+
160+
.. code-block:: python
161+
162+
{
163+
"test/test_something.py::test_form_upload[chrome]" : [
164+
"test/test_something.py::test_form_upload[image-chrome]",
165+
"test/test_something.py::test_form_upload[video-chrome]"
166+
],
167+
"test/test_something.py::test_form_upload[firefox]": [
168+
"test/test_something.py::test_form_upload[image-firefox]",
169+
"test/test_something.py::test_form_upload[video-firefox]"
170+
],
171+
"test/test_something_else.py::test_form_upload[firefox]": [
172+
"test/test_something_else.py::test_form_upload[image-firefox]",
173+
"test/test_something_else.py::test_form_upload[video-firefox]"
174+
],
175+
"test/test_something_else.py::test_form_upload[chrome]": [
176+
"test/test_something_else.py::test_form_upload[image-chrome]",
177+
"test/test_something_else.py::test_form_upload[video-chrome]"
178+
]
179+
}
180+
181+
You can also fall back on one of the default load distribution mechanism by passing the
182+
arguments for them listed above when you call pytest. Because this example returns
183+
``None`` if the nodeid doesn't meet any of the criteria, it will defer to whichever
184+
mechanism you chose. So if you passed ``--dist=loadfile``, tests would otherwise be
185+
divided up by file name.
186+
187+
Keep in mind, this is a means of optimization, not a means for determinism.
188+
189+
Controlling test group execution order
190+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
191+
192+
Sometimes you may want to have certain test groups start before or after others. Once
193+
the test groups have been determined, the ``OrderedDict`` they are stored in can have
194+
its order modified through the ``pytest_xdist_order_test_groups`` hook. For example, in
195+
order to move the test group named ``"groupA"`` to the end of the queue, this can be
196+
done:
197+
198+
.. code-block:: python
199+
200+
def pytest_xdist_order_test_groups(workqueue):
201+
workqueue.move_to_end("groupA")
92202
93-
* ``--dist=loadfile``: tests will be grouped by file name, and then will be sent to an available
94-
worker, guaranteeing that all tests in a group run in the same worker. This feature was added
95-
in version ``1.21``.
203+
Keep in mind, this is a means of optimization, not a means for determinism or filtering.
204+
Removing test groups from this ``OrderedDict``, or adding new ones in after the fact can
205+
have unforseen consequences.
96206

207+
If you want to filter out which tests get run, it is recommended to either rely on test
208+
suite structure (so you can target the tests in specific locations), or by using marks
209+
(so you can select or filter out based on specific marks with the ``-m`` flag).
97210

98211
Making session-scoped fixtures execute only once
99212
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Diff for: ‎changelog/18.feature.rst

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Allow defining of custom logic for test distribution among test groups, and changing the order in which test groups are passed out to workers.

Diff for: ‎src/xdist/newhooks.py

+52
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,55 @@ def pytest_xdist_node_collection_finished(node, ids):
5555
@pytest.mark.firstresult
5656
def pytest_xdist_make_scheduler(config, log):
5757
""" return a node scheduler implementation """
58+
59+
60+
@pytest.mark.trylast
61+
def pytest_xdist_set_test_group_from_nodeid(nodeid):
62+
"""Set the test group of a test using its nodeid.
63+
64+
This will determine which tests are grouped up together and distributed to
65+
workers at the same time. This will be called for every test, and whatever
66+
is returned will be the name of the test group that test belongs to. In
67+
order to have tests be grouped together, this function must return the same
68+
value for each nodeid for each test.
69+
70+
For example, given the following nodeids::
71+
72+
test/test_something.py::test_form_upload[image-chrome]
73+
test/test_something.py::test_form_upload[image-firefox]
74+
test/test_something.py::test_form_upload[video-chrome]
75+
test/test_something.py::test_form_upload[video-firefox]
76+
test/test_something_else.py::test_form_upload[image-chrome]
77+
test/test_something_else.py::test_form_upload[image-firefox]
78+
test/test_something_else.py::test_form_upload[video-chrome]
79+
test/test_something_else.py::test_form_upload[video-firefox]
80+
81+
In order to have the ``chrome`` related tests run together and the
82+
``firefox`` tests run together, but allow them to be separated by file,
83+
this could be done::
84+
85+
def pytest_xdist_set_test_group_from_nodeid(nodeid):
86+
browser_names = ['chrome', 'firefox']
87+
nodeid_params = nodeid.split('[', 1)[-1].rstrip(']').split('-')
88+
for name in browser_names:
89+
if name in nodeid_params:
90+
return "{test_file}[{browser_name}]".format(
91+
test_file=nodeid.split("::", 1)[0],
92+
browser_name=name,
93+
)
94+
95+
This would then defer to the default distribution logic for any tests this
96+
can't apply to (i.e. if this would return ``None`` for a given ``nodeid``).
97+
"""
98+
99+
@pytest.mark.trylast
100+
def pytest_xdist_order_test_groups(workqueue):
101+
"""Sort the queue of test groups to determine the order they will be executed in.
102+
103+
The ``workqueue`` is an ``OrderedDict`` containing all of the test groups in the
104+
order they will be handed out to the workers. Groups that are listed first will be
105+
handed out to workers first. The ``workqueue`` only needs to be modified and doesn't
106+
need to be returned.
107+
108+
This can be useful when you want to run longer tests first.
109+
"""

0 commit comments

Comments
 (0)