You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
for compatibility with Python 3.10.
- add `pytest >= 4.6.11` to argument `tests_require` of
the function `setuptools.setup` in script `setup.py`
This lower bound has been selected to ensure compatibility
with both Python 3 and Python 2.7. See below for details.
- add `pytest >= 4.6.11` to file `requirements.txt`
- remove `nose` from argument `tests_require` of
the function `setuptools.setup` in script `setup.py`
- remove `nose` from file `requirements.txt`
- update file `.travis.yml`
- remove the collection of coverage measurements on Travis CI,
because coverage measurement is incorrect (lower than the
real coverage, and even lower after switching to `pytest`)
due to Cython coverage not being correctly collected with
the build configuration currently used for testing.
Also, negligible changes in coverage measurements affected
the status of commits on GitHub, turning it to a red "X",
which can give the false impression that the tests failed
when they passed, and requires clicking on the "X" in order
to see more information that clarifies that the tests
actually passed.
- remove `coveralls` from file `requirements.txt`,
because `coveralls` was used only on Travis CI.
- add a configuration file `tests/pytest.ini`
and include it in `MANIFEST.in`
- ignore `.pytest_cache/` in `.gitignore`
Motivation
==========
The change from [`nose == 1.3.7`](
https://pypi.org/project/nose/1.3.7/#history)
to [`pytest`](
https://pypi.org/project/pytest)
is motivated by compatibility with Python 3.10.
`nose` is incompatible with Python 3.10
=======================================
The package `nose`, which was used to run the tests of `dd`,
is not compatible with Python 3.10 (for details,
read the last section below). And `nose` is unmaintained.
(Also, `nose` uses the `imp` module from Python's standard library,
which [is deprecated](https://docs.python.org/3.10/library/imp.html),
so may be removed in some future Python version.)
Summary of transition to `pytest`
=================================
In summary, using `pytest` with the existing tests requires adding
a [configuration file `tests/pytest.ini`](
https://docs.pytest.org/en/latest/reference/customize.html#configuration-file-formats)
to tell `pytest` which functions, classes, and methods to collect tests
from (called "discovery" of tests).
The [parameter `--continue-on-collection-errors`](
https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags)
tells `pytest` to not stop in case any test module fails to import,
and continue with running the tests. The ability to run the tests when
some `dd` C extension modules are not installed is necessary.
After transitioning the tests to `pytest`, the tests have been confirmed
to run successfully:
- on Python 2.7 with `pytest == 4.6.11`, and
- on Python 3.9 with `pytest == 6.2.4`.
Failed attempts to use `unittest`
=================================
First, I tried to use [`unittest`](
https://docs.python.org/3/library/unittest.html)
(part of CPython's standard library).
For writing `dd` tests, `unittest` suffices.
For *discovering* the tests, `unittest` seems to require that tests be
methods of subclasses of the class `unittest.TestCase`.
This is not the case in the tests of `dd` tests.
Using `pytest` allows changing the test runner from `nosetests`
with minimal changes to the tests themselves.
Test discovery using `unittest` could possibly be implemented by adding
a file `tests/__init__.py`, and defining in that file a function
`load_tests`, following the [documentation of `unittest`](
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover).
In any case, it is simpler to use `pytest`, which requires only
a configuration file.
If `unittest` encounters an `ImportError` during collection of the tests
(i.e., when it tries to import test modules), then it stops. There does
not appear to be any way to tell `unittest` to continue and run the rest of
the test modules (those that *could* be imported).
Usage of `nose`
===============
The dependence on `nose` is minimal.
Only one function is used from `nose`:
the function `nose.tools.assert_raises`.
The function `assert_raises` is dynamically defined
[in the module `nose.tools.trivial`](
https://github.com/nose-devs/nose/blob/release_1.3.7/nose/tools/trivial.py#L32-L54)
by instantiating the class `unittest.TestCase`,
and setting `assert_raises` to equal the
[bound method `assertRaises`](
https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises)
of the instance of `TestCase`.
So the function `assert_raises` from `nose` is just a PEP8-compliant
binding for the method `unittest.TestCase.assertRaises`.
Reading the code of `unittest`:
- https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L156-L243
- https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L704-L735
it follows that the existing usage:
```python
with nose.tools.assert_raises(AssertionError):
foo(1, 2)
```
is equivalent to the following code
(the use of `AssertionError` here is just as an example):
```python
with unittest.TestCase().assertRaises(AssertionError):
foo(1, 2)
```
Replacing usage of `nose` with `pytest` in test code
====================================================
The [context manager `pytest.raises`](
https://docs.pytest.org/en/latest/reference/reference.html#pytest-raises)
is a [drop-in replacement](
https://en.wikipedia.org/wiki/Drop-in_replacement)
for the function `nose.tools.assert_raises`:
```python
with pytest.raises(AssertionError):
foo(1, 2)
```
Also, the tests can still be run with `nosetests` on Python versions
where `nose` is still available.
Replacing the test runner `nosetests` with `pytest`
===================================================
- `pytest` correctly recognized the test files by default
- `pytest` does not recognize by default methods of classes that do not
start with "Test" as test methods, even if the methods start with `test_`.
The configuration file is necessary to change this behavior of `pytest`
(in particular the command-line parameter `-k` did not seem to work
for classes). Relevant documentation:
- https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery
- https://docs.pytest.org/en/latest/example/pythoncollection.html#changing-naming-conventions
- https://docs.pytest.org/en/latest/how-to/nose.html
- https://docs.pytest.org/en/latest/reference/reference.html#confval-python_classes
- https://docs.pytest.org/en/latest/reference/reference.html#confval-python_functions
- https://docs.pytest.org/en/latest/reference/reference.html#confval-python_files
- The call `pytest tests/foo_test.py` imports the package `dd` from
`site-packages` (assuming that the module `foo_test.py` contains the
statement `import dd`). So the default behavior of `pytest` is as
desired.
In contrast, `nosetests tests/foo_test.py` imports the package `dd`
from
the local directory `dd/`, even though `dd` *is* installed under
`site-packages`.
In any case, `pytest` is called from within the directory `tests/`,
as was done for `nosetests`.
`python -m pytest tests/foo_test.py` and
`PYTHONPATH=. pytest tests/foo_test.py` both result in importing `dd`
from the local directory `dd/`. Relevant documentation:
- https://docs.pytest.org/en/latest/explanation/pythonpath.html#invoking-pytest-versus-python-m-pytest
- https://docs.pytest.org/en/latest/how-to/usage.html#invoke-python
As remarked above, the `pytest` [parameter `--continue-on-collection-errors`](
https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags)
needs to be used for running the tests when some of the C extension modules are
not installed, for example:
```
cd tests/
pytest -v --continue-on-collection-errors .
```
or, to also activate [Python's development mode](
https://docs.python.org/3/library/devmode.html):
```
cd tests/
python -X dev -m pytest -vvv --continue-on-collection-errors .
```
Colored output of test results from `pytest`
============================================
With `nose`, I used to use [`rednose`](https://pypi.org/project/rednose/)
for coloring test results, which was convenient.
`pytest` colors its output by default, no plugin is required.
This capability is an optional way of viewing test results,
so the coloring comparison is mentioned only for completeness.
Observations about `pytest`:
- shows colored source code that includes more source lines
- detects assertions that failed, and marks their source lines
- avoids the deprecated `imp` module (standard library) that `nose` uses
(and thus the associated `DeprecationWarning`)
https://docs.python.org/3.10/library/imp.html
- running the tests of `dd` with `pytest` revealed several
`DeprecationWarnings` that were previously hidden by `nose`
(these warnings were about invalid escape sequences due to
backslashes appearing in non-raw strings).
Further remarks
===============
[`pytest == 6.2.4`](https://pypi.org/project/pytest/6.2.4) is not
compatible with Python 2.7.
[`pytest == 4.6.11`](https://pypi.org/project/pytest/4.6.11/)
is the latest version of `pytest` that is compatible with Python 2.7
(released on June 5, 2020).
`pytest` specifies `python_requires` [PEP 345](
https://www.python.org/dev/peps/pep-0345/#requires-python),
[PEP 503](
https://www.python.org/dev/peps/pep-0503/):
- https://github.com/pytest-dev/pytest/blob/4.6.11/setup.cfg#L48
- https://github.com/pytest-dev/pytest/blob/5.0.0/setup.cfg#L43
So including `pytest>=4.6.11` in the file `requirements.txt` suffices to
install, on each Python version, the latest version of `pytest` that is
compatible with that Python version.
This simplifies testing on CI, and packaging.
In other words, conditional installations in the file `.travis.yml` are
not needed for `pytest`, neither conditional definition of `tests_require`
in the script `setup.py`.
This approach leaves implicit the upper bound on `pytest` in
`tests_require`. This upper bound is specified explicitly by
`pytest` itself, depending on the Python version of the interpreter.
It appears that `pip == 9.0.0` and `setuptools == 24.2.0` are required
to correctly implement `python_requires`:
- https://pip.pypa.io/en/stable/news/#v9-0-0
- https://setuptools.readthedocs.io/en/latest/history.html#v24-2-1
How replacing usage of `nose` with `unittest` would have looked like
====================================================================
A way to replace `nose` could have been to add a module `tests/utils.py`
containing:
```python
"""Common functionality for tests."""
import unittest
_test_case = unittest.TestCase()
assert_raises = _test_case.assertRaises
```
which is close to what `nose` does. The function `assert_raises` could
then be imported from the module `utils` in test modules, and used.
Using `pytest` avoids the need for this workaround.
Details about the incompatibility of `nose` with Python 3.10
============================================================
[`nose == 1.3.7`](https://pypi.org/project/nose/1.3.7/#history)
imports in Python 3.10 fine, but `nosetests` fails, due to imports
from the module `collections` of classes that have moved to the
module `collections.abc`.
Relevant information about CPython changes in `collections`
(removal of ABCs):
- python/cpython#23754
- https://bugs.python.org/issue37324
- https://docs.python.org/3.10/library/collections.abc.html#collections-abstract-base-classes
- Deprecation since Python 3.3, present until Python 3.9:
https://docs.python.org/3.9/library/collections.html#module-collections
- Removed in Python 3.10:
https://docs.python.org/3.10/whatsnew/3.10.html#removed
About skipping tests
====================
The decorator `unittest.skipIf` is recognized by `pytest`, and skipped
tests are correctly recorded and reported. In any case, note also
the `pytest` test-skipping facilities:
- https://docs.pytest.org/en/latest/how-to/skipping.html
- https://docs.pytest.org/en/latest/how-to/unittest.html
- https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option
About passing `-X dev` to `python` in the `Makefile`
====================================================
The argument `dev` is available for the `python` option
[`-X` only on Python 3.7 and higher](
https://docs.python.org/3/library/devmode.html#devmode).
So the `Makefile` rules where `-X dev` appears are
not compatible with ealier Python versions supported by `dd`.
This is not an issue: the development environment is
intended to be Python 3.9 or higher, so there is no issue
with using `-X dev`.
Avoiding interaction between tests via class attributes
=======================================================
Avoid class attributes in test classes. Use [data attributes][1]
instead. Initialize the data attributes in setup methods of the
test classes, as is common practice. This approach avoids
interaction (via class attributes) between test scripts that
import the same modules of common tests.
With `nose`, this kind of interaction apparently did not
occur, as observed by test failures that were expected to happen.
However, `pytest` apparently runs tests in a way that
changes to imported modules (e.g., class attributes)
persist between different test scripts.
This `pytest` behavior was observed by the disappearance of
test failures when running with `pytest` (the test failures
were observable with `pytest` only when telling `pytest` to
run individual test scripts, instead of collecting tests from
all test scripts.
The cause of the issue with `pytest` was the modification of
class attributes (not [data attributes][1]) from the importing
module, of classes in the imported module. The modifications
were done by setting the class attribute `DD` that defines the
BDD or ZDD manager class. Both the scripts `cudd_test.py` and
`autoref_test.py` made modifications. The end result was
`autoref` tests being run using the class `dd.cudd.BDD`.
Using data attributes instead of class attributes,
and subclassing, avoids this kind of erroneous testing.
This approach is explicit [PEP 20][4].
Also, note that the `pytest` extension packages
[`pytest-xdist`][2] and [`pytest-forked`][3]
do not avoid this issue.
[1]: https://docs.python.org/3/tutorial/classes.html#instance-objects
[2]: https://github.com/pytest-dev/pytest-xdist
[3]: https://github.com/pytest-dev/pytest-forked
[4]: https://www.python.org/dev/peps/pep-0020/
0 commit comments