CI: add job that that uses pytest-dev version

[bsipocz]

As far as I see there is no CI job that tests using the dev version of the dependencies. My experience while maintaining a couple of pytest plugins is that having a job dor dev-dependencies is a very valuable smoke test to see when things start go wrong due to upstream changes.

(And I'm opening this issue as I discovered our own test suite at pytest-doctestplus is throwing a lot of failures when pytest-asyncio is installed and we use the dev version of pytest. I don't see any problems when pytest-asyncio is not present, so my best assumptions without diving into the rabbit hole, is that at least part of the failures are not doctestplus related)

[bsipocz]

OK, so all the failures are going away if/when I have the following ignore for the deprecation, so I suspect the problem itself is very much related to #924 and related issues.
Nevertheless, I would still highly recommend testing with pytest dev version and do run those test regularly from cron, too.

ignore:The configuration option "asyncio_default_fixture_loop_scope":pytest.PytestDeprecationWarning

[seifertm]

Thanks for the issue report!
I fully agree that pytest-asyncio should add a test environment that uses pre-release versions of its dependencies.

A first step towards this might be adding a new tox environment that uses default/requirements.txt, but ignores the constraints and adds the --pre flag to the pip install command. After this tox environment is available, we can integrate it as a non-required check in the delivery pipeline.

[bsipocz]

I would go further than --pre and would install from the github repo as they only do rc for major releases.

[webknjaz]

and adds the --pre flag to the pip install command

This would increase the likelihood of instability because it applies to everything in the deptree.

Instead, doing pytest >= 0.dev0 would only enable in for pytest but not other projects in the tree.

[webknjaz]

I would go further than --pre and would install from the github repo as they only do rc for major releases.

That's a good idea. Use the .tar.gz URL so calling Git isn't even necessary.

[webknjaz]

After this tox environment is available, we can integrate it as a non-required check in the delivery pipeline.

One option could be only running it on cron (and perhaps merge queues) and not PRs to avoid confusing the contributors.

[webknjaz]

@seifertm by the way, as long as pytest IS somewhere in tox deps, it's possible to just use --force-dep in CI: https://github.com/sphinx-contrib/sphinxcontrib-towncrier/blob/53943d5/.github/workflows/ci-cd.yml#L510.

[bsipocz]

One option could be only running it on cron (and perhaps merge queues) and not PRs to avoid confusing the contributors.

Yes, that's certainly a way, and works really well when also coupled with a labelling system that can trigger these non-default jobs (I suppose this much scaffolding is an overkill for a plugin though).
I've seen both approaches, for smaller packages where one doesn't really have many inexperienced new contributors having the dev job be part of the PR process, but elsewhere it's only cron triggered.

Here is another example tox where we don't even have a dev env for multiple dependencies, just doing it for pytest: https://github.com/scientific-python/pytest-doctestplus/blob/main/tox.ini#L34
But this one have multiple upstream libraries, some of which provide nightly wheel builds, too: https://github.com/astropy/astroquery/blob/main/tox.ini#L34