Improve test structure for various typechecker

[UnknownPlatypus]

When working on #3317, I would have liked to be able to run pyright/ty/pyrefly on some yaml test case we have because some feature that previously required mypy plugin now should work with builtin python typing constructs. There are also preliminary support for django feature added in pyrefly for ex that we could test here.

So I was wondering how we could handle typechecking tests in tests/typecheck against more typechecker.

Current state

We currently use:

• pytest-mypy-plugins for tests in tests/typecheck but it only support mypy currently.
• assert_type tests in tests/assert_type running against mypy / pyright / pyrefly & ty

We already migrated some tests to assert_type when they don't required mypy plugin to extand coverage but it get's a bit more complicated once we touch django models.

Migration options

I think it would be nice to gradually move some tests from tests/typecheck to assert_type to get more coverage but there are still some usefull feature in pytest-mypy-plugins that we might lose along the way.

• registration of models for django via installed_apps
• assertion of detailed error output
• custom django settings

There are a few directions I see and would like to gather feedback and ideas to solve this:

• Migrate to assert_type.
  • We would need a directory structure convention to register models. We can take inspiration from django own test suite that does that with a bunch of submodules automatically registered as django apps by the test runner.
  • We would lose precise error message assertion
• Extend pytest-mypy-plugins with multi-checker backends
  • I suspect this would be a huge effort because the package was designed with only mypy in mind
  • We would have to find a way to assert error messages from other typecheckers too
• pytest-typing
  • Similar to the assert_type test we have but a bit easier to digest maybe, it's inspired from ty typing test framework
  • Allows asserting exact error messages too
• Another idea ?

I tried a small poc for the Migrate to assert_type option by migrating ŧest_base.yml, it mostly work but we do loose the exact error assertions

What do you guys think ? Any idea on how to make this easier ?

[ngnpope]

Thanks for investigating this 🙌🏻

Firstly, I think that we should try to move away from pytest-mypy-plugins. While it has served us well until now, it's a bit challenging to work with - not least because the content of the files embedded in the YAML cannot be easily formatted ¹. It would take a lot of effort to make it support multiple type checkers - as well as a rename 😅

As a first thought, for very simple things, assert_type() is quite nice as it requires no dependencies and works for all type checkers. There is also the advantage that a base set of models can be shared across multiple tests reducing the amount of code to maintain. The loss of explicit assertions of error messages is a negative though ².

However, I think that we should steer toward using pytest-typing. It looks great! The support for asserting exact error messages and running against all the major type checkers³ is great. The main thing to confirm is that it supports multi-file tests as we'd need that to set up models properly.

Being based on Astral's mdtest framework, we get some great benefits:

• The tests become documentation explaining what the test is showing - something that's lacking in our current approach. This will be particularly handy for documenting expected behaviour for some of the more complex areas in the stubs, as well as specific behaviour implemented by the plugin. As regards the latter, the ability to target tests as specific type checkers is useful.
• Because the code is within code fences in the markdown files, ruff's formatter is able to format the code resulting in cleaner, more consistent tests and docs.

In addition, it won't be too hard for Astral to hit the ground running when implementing their built-in support for Django (to provide the capabilities that our plugin provides for mypy). They'll be largely familiar with the format and could likely help expand the test-suite here to benefit all type checkers.

Footnotes

1. I thought about writing something to do that, but we can avoid that effort as will be explained. ↩

2. One thing I do plan to do is make sure that django-stubs is not using misc for everything - we really should define custom error codes! Hopefully coming soon if I get some time in the next few weeks. This would somewhat offset the loss of more detailed assertions by avoiding generic # type: ignore[misc] everywhere. ↩

3. Except zuban... Maybe we can get that added too. (It has some preliminary support for Django.) ↩

[sobolevn]

Yes, I agree: our goal should be to have an uniform testing interface for all the type-checkers we support. pytest-mypy-plugins only works for mypy :(

[UnknownPlatypus]

One thing I do plan to do is make sure that django-stubs is not using misc for everything - we really should define custom error codes! Hopefully coming soon if I get some time in the next few weeks. This would somewhat offset the loss of more detailed assertions by avoiding generic # type: ignore[misc] everywhere

Totally agree, tho I'm wondering if it might cause pain for users doing incremental typing with a lot of type:ignore to update

Except zuban... Maybe we can get that added too. (It has some preliminary support for Django.)

I'm all for adding it too, but maybe after the work in #3317 land to avoid too much conflict

As a first thought, for very simple things, assert_type() is quite nice as it requires no dependencies and works for all type checkers. There is also the advantage that a base set of models can be shared across multiple tests reducing the amount of code to maintain. The loss of explicit assertions of error messages is a negative though 2.

I did that for #3317 because I needed a way to check against the other typechecker. I went for a very minimal approach using the folder structure to declare django apps and model for when the mypy plugin requires it.

I think it's a good first step but would very like something like pytest-typing. My main concern is also about performance because I believe it's spawns a bunch of mypy like pytest-mypy-plugins does which has always been kinda slow. The simple approach with assert_type and regular files makes it very fast