TokenStream.push()

[rob-ross]

In both your projects TokenStream has a push() method that adds the argument token to a queue, then the next time next() is called, it fetches from the top of the queue first if it has any values. And the push() algorithm is

1. put the current token in the queue
2. make the argument token the new current token.

This would allow any arbitrary token to be inserted in the parser processing, even one not originally emitted by the Lexer.
However, all calls to push() use the current token, i.e., push(stream.current). This has the effect that the next time next() is called, you'll get back that token again. It's almost like backtracking one token in the stream, except the current token wouldn't be the previous value after backtracking, it would remain the current token.

So what is the difference between this idiom, and actually backtracking one Token in the stream? The next code to call current() will get a token, but the next call to next() will return that same token.

[rob-ross]

In both versions, at the end of parse_path()/parse_query(), you have

            else:
                if in_filter:
                    stream.push(stream.current)
                break

            stream.next_token()

which terminates the while True: loop
If in_filter is True, you backtrack one token. Then regardless of the flag, you advance the stream. Would this code be equivalent to:

else:
    if not in_filter:
        stream.next_token()
    break

Or is there a subtlety that I am missing here?

Edit:

Yes, there is a subtley I am missing. I tried this change and the unit tests failed. They hung in this method, so it needs that next_token() to proceed.

[jg-rp]

Those TokenStream classes were copied from another project of mine and adapted for JSONPath parsing. In the original project it was important to be able to push arbitrary tokens in to the stream and to step through an iterator without consuming it because the number of tokens could be huge.

That is not the case for JSONPath queries, and this is another example of where following the v2 branch is preferable.

Notice that in this version of TokenStream we consume the iterator during initialization and maintain an index to the current token. A backup operation is then simply a stream.pos -= 1.

python-jsonpath/jsonpath/stream.py

Lines 16 to 20 in 7e09eaf

	def __init__(self, token_iter: Iterable[Token]):
	self.tokens = list(token_iter)
	self.pos = 0
	path = self.tokens[0].path if self.tokens else ""
	self.eof = Token(TOKEN_EOF, "", -1, path)

And here there's no in_filter argument because all parse_* methods are expected to consume their last token rather than leave it on the stream.

python-jsonpath/jsonpath/parse.py

Line 332 in 7e09eaf

def parse_query(self, stream: TokenStream) -> Iterable[JSONPathSegment]:

There are still places where I "put the last token back on the stream", but this is done for efficiency (in terms of lines of code rather than execution performance) and could be avoided with extra "peeking" or using current() more often than next().