Generate more tokens, especially within tags

[lxcid]

• New Line
• Whitespace
• Dot
• Parentheses
• Comma

[lxcid]

Proposal:

Right now, scan and scanUntil of Scanner provide the bulk of lexer logic but it doesn't tokenize new line, white spaces, dot, parentheses and comma which can greatly empower our parser.

Mustache lexer can't be context free as its delimiter can be changed on run time. So the lexer need to be able to identify tags (denoted by wrapping in open and close delimiters) and look out for set delimiter tag. Triple mustache {{{ can also complicate the lexer logic.

The goal is to:

• Store only range in lexer token.
• able to parse delimiter and response to set delimiter.
• handle open and close escaped tag modifier.
• We should only visit each character only once.