Failed TypeVar substitution by asynccontextmanager

[BoniLindsley]

Bug Report

Return type of contextlib.asynccontextmanager does not substitute typing.TypeVar determined from parameters.

To Reproduce

Pass the following script to mypy, using mypy --config-file= main.py or otherwise.

#!/usr/bin/env python3

import contextlib, typing

_T = typing.TypeVar('_T')


@contextlib.asynccontextmanager
async def identity(element: _T) -> typing.AsyncIterator[_T]:
    """
    A context manager that does nothing
    and returns the given element.
    Type checking should assume the yielded value
    has the same type as the parameter.
    """
    yield element


async def main() -> None:
    """
    Here we give an ``int`` to ``identity``.
    So the yielded ``number`` should be ``int`` as well.
    """
    async with identity(1) as number:
        """Problem: We got ``_T`-1`` as its type instead of ``int``."""
        reveal_type(number)
        number = 2

Expected Behavior

The type of number should be int. In particular, assignment of the integer 2 to it should be compatible, and so mypy should return without errors.

Actual Behavior

The variable number is assigned, what I assume, is the TypeVar. This leads to an incompatible assignment of 2.

main.py:26: note: Revealed type is '_T`-1'
main.py:27: error: Incompatible types in assignment (expression has type "int", variable has type "_T")
Found 1 error in 1 file (checked 1 source file)

As a side note, a non-async counterpart of this behaves as expected, and does not produce any errors.

Your Environment

• Mypy version used: 0.790
• Mypy command-line flags: --config-file=
• Mypy configuration options from mypy.ini (and other config files): Not applicable.
• Python version used: Python 3.9.1
• Operating system and version: Debian bullseye/sid and Windows 10 version 1909 build 18363.1256

[JelleZijlstra]

This might be a typeshed bug; relevant definitions are for @asynccontextmanager (https://github.com/python/typeshed/blob/master/stdlib/2and3/contextlib.pyi#L32) and AsyncContextManager (https://github.com/python/typeshed/blob/master/stdlib/3/typing.pyi#L415). I don't see anything obviously wrong in those stubs though.

[hauntsaninja]

Note that mypy currently special cases contextlib.contextmanager (but doesn't special case contextlib.asynccontextmanager):

mypy/mypy/plugins/default.py

Line 172 in 92923b2

def contextmanager_callback(ctx: FunctionContext) -> Type:

This is something PEP 612 should help with.

[erictraut]

I think mypy is doing the right thing here given the current definition of asynccontextmanager in contextlib.pyi. As @hauntsaninja mentioned, it will need to be updated to make use of ParamSpec (defined in PEP 612).

I confirmed that it works as expected if the typeshed definition of asynccontextmanager is changed to:

_P = ParamSpec("_P")
def asynccontextmanager(func: Callable[_P, AsyncIterator[_T]]) -> Callable[_P, AsyncContextManager[_T]]: ...

We need to wait for python/typeshed#4827.

[BoniLindsley]

I looked into how contextmanger is handled, and I think its special casing can be applied to asynccontextmanager without change. Literally a two-line fix to bring it over. (I am unsure of the if-tree order for optimisation purposes though.) May be it can be a temporary fix until mypy has PEP612 compliance?

I tried to add a regression test, by copying over the non-async version, but it seems to fail, unable to find asynccontextmanager in contextlib. I am missing something obvious, I think. A little hint would be appreciated.

A unit test using minimal stubs works though. And I believe this test more accurately reflects the issue.

Please let me know what to do from here, if this fix can be used.

[BoniLindsley]

PEP 612 targets Python 3.10. Does that mean special casing for < 3.10 will still be necessary?

I would like to submit a temporary fix for it, but I am having trouble getting a unit test to work. Specifically, a test in test-data/unit/pythoneval.test. According to its comment, its tests use full stubs. But the test I add to it is not able to find contextlib.asynccontextmanager. Here is a summary of what I did, and it should run as is.

#!/usr/bin/bash

# Check out into current directory if not already done.
if [ -z "$(ls -A .)" ]; then
  git clone [email protected]:BoniLindsley/mypy.git .  #
fi

# Make sure things are up to date.
git submodule update --init
python3 -m pip install --upgrade --requirement test-requirements.txt

# Add test if not already done.
test_file=test-data/unit/pythoneval.test
if ! grep testAsyncContextManager $test_file > /dev/null; then
  cat << EOF >> test-data/unit/pythoneval.test

[case testAsyncContextManager]
# flags: --python-version 3.7
import contextlib
from contextlib import asynccontextmanager
from typing import AsyncIterator

@asynccontextmanager
async def f(x: int) -> AsyncIterator[str]:
    yield 'foo'

@contextlib.asynccontextmanager
async def g(*x: str) -> AsyncIterator[int]:
    yield 1

reveal_type(f)
reveal_type(g)

with f('') as s:
    reveal_type(s)
[out]
_program.py:13: note: Revealed type is 'def (x: builtins.int) -> typing.AsyncContextManager[builtins.str*]'
_program.py:14: note: Revealed type is 'def (*x: builtins.str) -> typing.AsyncContextManager[builtins.int*]'
_program.py:16: error: Argument 1 to "f" has incompatible type "str"; expected "int"
_program.py:17: note: Revealed type is 'builtins.str*'
EOF
fi

# The actual test.
python3 -m pytest --quiet -k testAsyncContextManager

The test is expected to fail, since this does not include the fix, and to produce a diff with wrong revealed types. But the error from it is instead

Actual:
  _testAsyncContextManager.py:3: error: Module 'contextlib' has no attribute 'asynccontextmanager'; maybe "contextmanager" or "AsyncContextManager"? (diff)
  _testAsyncContextManager.py:10: error: Module has no attribute "asynccontextmanager"; maybe "contextmanager" or "AsyncContextManager"? (diff)

So, the question is, how do I make the test find asynccontextmanager? May be I am missing something obvious.

Environment: Debian, pyenv, Python 3.9.1.