Skip to content

Crash with "Should never get here in normal mode" with incremental=True #3852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lordmauve opened this issue Aug 21, 2017 · 9 comments
Closed

Crash with "Should never get here in normal mode" with incremental=True #3852

lordmauve opened this issue Aug 21, 2017 · 9 comments
Assignees
@ilevkivskyi
Labels
crash

Comments

@lordmauve
Copy link

I was working in a file, adding types and replacing attrs with typing.NamedTuple in order to benefit from types, when I got this crash:

$ bin/mypy python/
Traceback (most recent call last):
  File "bin/mypy", line 33, in <module>
    sys.exit(console_entry())
  File "dist/lib/python3.6/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "dist/lib/python3.6/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "dist/lib/python3.6/mypy/main.py", line 97, in type_check_only
    options=options)
  File "dist/lib/python3.6/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "dist/lib/python3.6/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "dist/lib/python3.6/mypy/build.py", line 2037, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "dist/lib/python3.6/mypy/build.py", line 2106, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "dist/lib/python3.6/mypy/build.py", line 1547, in fix_cross_refs
    self.manager.options.quick_and_dirty)
  File "dist/lib/python3.6/mypy/fixup.py", line 22, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "dist/lib/python3.6/mypy/fixup.py", line 105, in visit_symbol_table
    self.visit_type_info(value.node)
  File "dist/lib/python3.6/mypy/fixup.py", line 62, in visit_type_info
    base.accept(self.type_fixer)
  File "dist/lib/python3.6/mypy/types.py", line 435, in accept
    return visitor.visit_instance(self)
  File "dist/lib/python3.6/mypy/fixup.py", line 176, in visit_instance
    assert self.quick_and_dirty, "Should never get here in normal mode"
AssertionError: Should never get here in normal mode

My mypi.ini contained

[mypy]
follow_imports = silent
incremental = True

However the error went away when I removed incremental = True.
@gvanrossum
Copy link
Member

Thanks for the report -- if you can still repro this, is your source code public, and can you save a tarball of your .mypy_cache?

@lordmauve
Copy link
Author

I'm afraid it's not public.

@mbforbes
Copy link

mbforbes commented Sep 26, 2017 •
edited
Loading

I'm also seeing this bug. (I also have another, possibly related bug, which I'll report in a separate comment below.)

Disclaimer: I'm pointing MYPYPATH to site-packages/ (many details in #3987), which I believe is unsupported.

I've made my code open source at mbforbes/rndjam1 @ 4714532. If you're trying to reproduce this and are having trouble running it, feel free to reply to me here, open an issue there, etc.

This error only occurs for me the second time I run mypy with incremental mode, which I believe means it has to build the cache first before this problem happens. I need to introduce a type error for it to happen. And if I comment out one of my local imports (dataio in my code), the error goes away.

For context:

$ rm -rf .mypy_cache/

$ mypy --version
mypy 0.530-dev-c332ea08b542728985fadcbee1db6651060c3876

$ cat mypy.ini 
[mypy]
mypy_path=/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/

Then:

$ mypy --ignore-missing-imports --follow-imports=silent --incremental regression.py 
regression.py:206: error: Argument 1 to "least_squares" has incompatible type "int"; expected "FloatTensor"

$ mypy --ignore-missing-imports --follow-imports=silent --incremental regression.py 
Traceback (most recent call last):
  File "/home/max/.pyenv/versions/rndj1/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.530-dev-c332ea08b542728985fadcbee1db6651060c3876', 'console_scripts', 'mypy')()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1838, in dispatch
    process_graph(graph, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2081, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2150, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1576, in fix_cross_refs
    self.manager.options.quick_and_dirty)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 22, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 105, in visit_symbol_table
    self.visit_type_info(value.node)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 62, in visit_type_info
    base.accept(self.type_fixer)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 476, in accept
    return visitor.visit_instance(self)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 176, in visit_instance
    assert self.quick_and_dirty, "Should never get here in normal mode"
AssertionError: Should never get here in normal mode

If I keep running the same command, I keep getting the error.

I've attached the result of tar -czvf mpc.tar.gz .mypy_cache/. mpc.tar.gz

(Edit: linked to specific commit in repository.)

@mbforbes
Copy link

With exactly the same setup as above, I get a different error if I run with --quick-and-dirty instead of --incremental:

$ mypy --ignore-missing-imports --follow-imports=silent --quick-and-dirty regression.py 
regression.py:206: error: Argument 1 to "least_squares" has incompatible type "int"; expected "FloatTensor"

$ mypy --ignore-missing-imports --follow-imports=silent --quick-and-dirty regression.py 
Traceback (most recent call last):
  File "/home/max/.pyenv/versions/rndj1/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.530-dev-c332ea08b542728985fadcbee1db6651060c3876', 'console_scripts', 'mypy')()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1838, in dispatch
    process_graph(graph, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2081, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2148, in process_fresh_scc
    graph[id].load_tree()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1570, in load_tree
    self.tree = MypyFile.deserialize(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 249, in deserialize
    tree.names = SymbolTable.deserialize(data['names'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2435, in deserialize
    st[key] = SymbolTableNode.deserialize(value)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2382, in deserialize
    node = SymbolNode.deserialize(data['node'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 182, in deserialize
    return method(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 674, in deserialize
    type = None if data['type'] is None else mypy.types.deserialize_type(data['type'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 30, in deserialize_type
    return method(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1375, in deserialize
    return TypeType.make_normalized(deserialize_type(data['item']))
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1356, in make_normalized
    return TypeType(item, line=line, column=column)  # type: ignore
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1344, in __init__
    if isinstance(item, CallableType) and item.is_type_obj():
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 721, in is_type_obj
    return t is not None and t.is_metaclass()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2250, in __getattribute__
    raise AssertionError('De-serialization failure: TypeInfo not fixed')
AssertionError: De-serialization failure: TypeInfo not fixed

@ilevkivskyi
Copy link
Member

@mbforbes Interesting, this points to some familiar places in code. I will take care of this.

@ilevkivskyi ilevkivskyi self-assigned this Sep 26, 2017
This was referenced Sep 30, 2017
Merged
Merged
@ilevkivskyi
Copy link
Member

ilevkivskyi commented Oct 2, 2017 •
edited
Loading

@gvanrossum I have a simple repro inspired by #4043 for the first crash here as well (the second is unrelated and will be fixed by #4038). Here is the recipe:

Initial setup:

# a.py
from b import x

# b.py
from c import C
x: C

# c.py
class C: pass

with any previous mypy caches removed. Steps to reproduce:

  • run mypy a.py
  • replace the content of c.py with def C(): pass
  • run mypy -i a.py (it will show an obvious error)
  • add random whitespace to a.py
  • run mypy -i a.py again -- AssertionError: Should never get here in normal mode

@gvanrossum
Copy link
Member

Interestingly, this one was not (directly) introduced by #2014. Bisecting points to #3304 (Tighten FakeInfo and fix crashes in --quick mode; rev d2638b1).

@gvanrossum
Copy link
Member

the second is unrelated and will be fixed by #4038

This refers to the crash reporting AssertionError: De-serialization failure: TypeInfo not fixed, right?

@ilevkivskyi ilevkivskyi mentioned this issue Oct 2, 2017
Merged
@ilevkivskyi
Copy link
Member

ilevkivskyi commented Oct 2, 2017 •
edited
Loading

the second is unrelated and will be fixed by #4038

This refers to the crash reporting AssertionError: De-serialization failure: TypeInfo not fixed, right?

Yes.

gvanrossum pushed a commit that referenced this issue Oct 4, 2017
@ilevkivskyi @gvanrossum
Delete cache for a module if errors are found (#4045)
51fb765 
Fixes #4043
Fixes #3852
Fixes #3052

This is a simple-minded solution for inconsistent cache state problem discovered in #4043.
JukkaL pushed a commit that referenced this issue Oct 11, 2017
@ilevkivskyi @JukkaL
Fix --incremental crash on Type[...] (#4038)
bd1b3d7 
This fixes the second traceback reported in #3852. This also improves member 
lookup on `Type[...]` by moving it to a later stage (`checkmember.py` instead 
of some manipulations in `TypeType` constructor that are dangerous during 
de-serialization).

Also fixes the first case in #4058.
@pradyunsg pradyunsg mentioned this issue Jun 11, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ilevkivskyi ilevkivskyi

Labels
crash
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lordmauve @mbforbes @gvanrossum @ilevkivskyi