Add pyright to tests

[srittau]

I propose to add pyright to our CI test suite. pyright is a stable type checker and probably the fastest to implement new typing features. Also:

• pyright is frequently used as it is the standard type checker for the popular Visual Studio Code IDE/editor.
• pyright has many useful warnings, not found in other type checkers.
• Not running pyright as part of our CI complicates the job for pyright's maintainers as evidenced by the frequent clean-up PRs by Eric Traut. This will be even more of an issue when pyright decides not to ship third-party stubs.

I think the discussion in #5035 shows that adding pyright to CI is uncontroversial. Cc'ing @JelleZijlstra @hauntsaninja and @erictraut.

[srittau]

(And of course by "pylance" I meant "pyright". Corrected it above.)

[erictraut]

I think this is a great idea. If there are no objections, we (the pyright contributors) will work on a PR that incorporates the command-line version of pyright into a CI test for typeshed. We'll let you know if we need help with any of the specifics.

[srittau]

@erictraut FWIW I have taken a stab at a general framework for this in #5051. I'd be glad if your team takes over, and you can use elements from that PR as you feel appropriate. The main thing I'd like to see is that it keeps the general idea of having a tests/pyright_test.py script to run the tests. The test should also support a tests/pyright_exclude_list.text file, like the other type checkers do.

[erictraut]

That's a great start. Thanks!

Pyright uses a configuration file called "pyrightconfig.json" to control which diagnostic rules are enabled, which directories should be included/excluded, etc. This config file is expected to be at the root directory of the project. Would it be OK if we added this file at the root of typeshed and used that instead of a text-based exclude list? If that isn't acceptable, we could auto-generate the pyrightconfig.json file as part of the test script.

[srittau]

Pyright uses a configuration file called "pyrightconfig.json" to control which diagnostic rules are enabled, which directories should be included/excluded, etc. This config file is expected to be at the root directory of the project. Would it be OK if we added this file at the root of typeshed and used that instead of a text-based exclude list? If that isn't acceptable, we could auto-generate the pyrightconfig.json file as part of the test script.

Sounds fine to me.

[srittau]

A small additional thought: tests/pyright_exclude_list.txt could contain a small note pointing to pyrightconfig.json so people unfamiliar with pyright know where to look.

[jakebailey]

I opened #5059 which is based on #5051 with the PR comments addressed, and will update it with a more finalized config.