PEP 702 (@deprecated): improve the handling of overloaded functions and methods

[tyralla]

Fixes #18323

Mypy's current approach to detecting deprecated overloads of functions and methods that I implemented has many drawbacks. For example, it does not handle some kinds of unions (see the discussion in #18477) and generic types (#18323). The complicating matter is that we made deprecated a symbol property instead of a type property (as we decided in #17476), but the selection of the relevant overload happens when only the types are reliably available. My first attempt was to search for possible deprecations at places where the symbols and types of all overloads and the type of the relevant overload are known. The naive assumption was that one could compare the relevant overload type with all available overload types to find the relevant overload symbol. However, the relevant overload type is often a modification of one or multiple of the original overload types. Hence, one would have to repeat all these possible modifications. #18323 would improve the current algorithm by adding one more of these modifications, but it seems worth trying a completely different way, which is what this PR is about.

The proposed solution is to handle the situation at the place where the relevant overload or overloads are detected by querying the relevant symbols from the available modules and assigning them to the overload types beforehand. Therefore, I use the already available but "unreliable" definition attribute. This approach requires including the deprecated attribute into symbol snapshots (relevant for caching), which might have been necessary anyhow (I have now added corresponding tests to the fine-grained test suite). It seems the approach works, but if it is risky in some way, I could soften it by resetting the previous definition values, updating only the deprecated subattributes instead of the complete definition attributes, adding a special-purpose attribute, or something like that.

Regarding the Mypy primer change for pydantic, I did not investigate why this change happens, but it seems like an improvement as the old notes seem really buggy ( pydantic/fields.py:542: note: def Field(default, default: EllipsisType, *...).

There is one use mypy.types.get_proper_type() error in both "Type check your own code" test suits. Either there is a bug, or I guess I am missing something very obvious...

I am interested to hear what you think about it!

Please note that I took the extension of the testDeprecatedDescriptor test case from #18333 by @Viicos.

[tyralla]

Oh no, I lost a few extensions of the check-deprecated test suite. I will re-add them later...

[tyralla]

Oh no, I lost a few extensions of the check-deprecated test suite. I will re-add them later...

fixed

[sobolevn]

This looks like a bug in mypy's plugin / source: if isinstance(c := get_proper_type(inferred_type), CallableType): is not detected properly by our proper_type plugin

[tyralla]

This looks like a bug in mypy's plugin / source: if isinstance(c := get_proper_type(inferred_type), CallableType): is not detected properly by our proper_type plugin

I was afraid of something like that. I have not dealt with this plugin before, but it does not look that complicated. I will try to fix it later.

[tyralla]

This looks like a bug in mypy's plugin / source: if isinstance(c := get_proper_type(inferred_type), CallableType): is not detected properly by our proper_type plugin

I was afraid of something like that. I have not dealt with this plugin before, but it does not look that complicated. I will try to fix it later.

I've got it. Mypy does not understand the following usage of zip:

mypy/mypy/checkexpr.py

Line 2696 in 5202c98

returns, inferred_types = zip(*unioned_return)

So, inferred_types (and returns) become Any, which seems to confuse the proper_plugin. Supporting this usage of zip and improving proper_plugin (provide a more useful error message) would be good, but for this PR I will just change the problematic line in the next commit.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:542: note:     def Field(default, default: EllipsisType, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> Any
+ pydantic/fields.py:542: note:     def Field(default: EllipsisType, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> Any
- pydantic/fields.py:542: note:     def [_T] Field(default, default: _T, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
+ pydantic/fields.py:542: note:     def [_T] Field(default: _T, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
- pydantic/fields.py:542: note:     def [_T] Field(default_factory, *, default_factory: Callable[[], _T] | Callable[[dict[str, Any]], _T], alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
+ pydantic/fields.py:542: note:     def [_T] Field(*, default_factory: Callable[[], _T] | Callable[[dict[str, Any]], _T], alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
- pydantic/fields.py:542: note:     def Field(alias, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> Any
+ pydantic/fields.py:542: note:     def Field(*, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: Any | None = ..., ge: Any | None = ..., lt: Any | None = ..., le: Any | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> Any

django-stubs (https://github.com/typeddjango/django-stubs): 2.05x faster (51.5s -> 25.1s in a single noisy sample)

freqtrade (https://github.com/freqtrade/freqtrade): 1.72x slower (129.0s -> 222.2s in a single noisy sample)