gh-140950: Reset shlex state in push_source() so post-EOF pushes are read

[LeSingh1]

Fixes gh-140950.

After shlex.shlex.get_token() reaches end of input it sets self.state to None. A subsequent push_source() did not reset that state, so the next get_token() call returned EOF immediately and the newly pushed source was silently ignored.

pop_source() already resets state to ' ' for the same reason. This change does the same in push_source(), which is the minimal symmetric fix.

Reproducer from the issue:

>>> from shlex import shlex
>>> lexer = shlex('a')
>>> lexer.get_token()
'a'
>>> lexer.get_token()      # EOF, state becomes None
''
>>> lexer.push_source('b')
>>> lexer.get_token()      # before: '' (EOF); after: 'b'
'b'

Push during an active stream is unaffected because state is already ' ' in that case. A regression test is added in Lib/test/test_shlex.py and a Library news entry is added under Misc/NEWS.d/next/Library/.

[python-cla-bot]

The following commit authors need to sign the Contributor License Agreement:

• sshaurya914@gmail.com