Replace RegEx based syntax with external AST parser

[alerque]

We keep mentioning this in various places, but apparently we don't have an issue to track it. Lets try to collect links to relevant tools and discussions to help move towards a syntax highlighter that actually understands Markdown instead of trying to fudge with Regular Expressions.

1. Neovim has an API for quickly replacing lots of highlights via an asynchronous background job. To date I don't think VIM has anything on par with this feature. Are we okay with a Neovim only improvement? Personally I'm okay with this because I exclusively use Neovim anyway. Since this might be a lot easier to implement with direct API access and VIM support could be added later, I'm totaly okay with putting my own time into Neovim only features. Of course the RegEx could be a fall-back, I wouldn't neuter the plugin for VIM users, only have a less nice set of features.

2. What, if any, plugin language should we use? Pure VimL might be ideal, but I am not as comfortable with it as Lua or Python, and it's likely much slower. Personally I think if we can make this work in Lua that should be our way forward.

3. What external parser should be actually use? Ideally this would be Pandoc of course, but it does not currently provide position information in its AST. Until this is implemented or CommonMark-hs is up to speed I don't think we have a practical way forward. I'm not enough of a Haskell ninja to even help on CommonMark yet. The best Lua based Markdown parser I know of is LuaMark (from the same author as Pandoc). Another option might be a filter that takes the Pandoc AST and walks it against the source to inject position information.

With those decisions made, here is a running to-do list of things I think we need to get there:

• Add a feature flag to toggle the new syntax highlighter so people can use the RegEx ones either by preference or because they don't have the tools. Note this will also make development easier as we can start hacking on it without an expectation of it working yet.
• Figure out the basic tooling for getting an external AST with source data injected into the syntax highlight system.
• ...

[alerque]

Issues this would fix, obsolete, or deprecate:

• Code highlighting fails in combination with reference link #299
• Set up commentring in the document's YAML metadata region #298
• Syntax highlighter is too slow #296
• Slow syntax highlighting profile #172
• Intra-word underscores need proper support #286
• Continuation indent for lists #161
• Inline links no longer highlighted when complete #281
• [](){target="_blank"} triggers emphasis. #290
• Regression in pandocReferenceDefinition #289
• Syntax error in tagged code block breaks highlighting outside the code block. #283
• Dollar signs not handled the way Pandoc handles them #284
• Italics display problem when trailing asterisk not followed by blank space #278
• Numbered list point with link and underscore causes issue with syntax highlighting #272
• Vim-pandoc mistakenly recognizes numbers followed by a period inside of a paragraph as a numbered list #277

... Nevermind. There are only 4 or 5 open issues not affected by this. I'm going to stop pinging them for now. I'll keep this comment as a place to keep notes on how each one is actually implemented because many will require special attention.

[alerque]

Other related references:

• https://github.com/numirias/semshi — Neovim + Python basaed highlighter for Python
• https://github.com/arakashic/chromatica.nvim — Neovim + Python based highlighter for C languages
• https://github.com/jeaye/color_coded — Lua/VimL/C++ based highlighter using libclang
• https://github.com/billyvg/tigris.nvim — Neovim + NodeJS based highlighter for Javascript.
• Look into using Lua PEG sile-typesetter/vim-sile#1 — Issue for tracking my desire to do roughly the same thing for SILE source files.

[fmoralesc]

Just some quick thoughts:

• I think creating an nvim only implementation of this might be the best shot, although vim 8 probably could do something similar also (I also use neovim exclusively, so that's my bias). I think the upcoming extended marks feature could be useful for this purpose as well. I had been waiting for this to look more into this problem.
• I am more comfortable in python than in lua, but both languages would be fine to move forward.
• If we plan to do this incrementally, we could start by creating a custom parser to detect blocks first (that could be reused by vim-pandoc for autoformatting already). We could then generalize it in a way that could be used for sile and other things as well.

Obviously, the ideal solution would be to use pandoc's full AST, but the metadata we need is still missing.

[alerque]

It looks like the tree-sitter branch still waiting merge into Neovim is what will largely enable this possibility on the editor side. See [https://github.com/neovim/neovim/issues/1767]. Of course we still need the other side of that too, but it does look like it is key.