use tree-sitter

[tpapp]

tree-sitter framework for incremental parsing of source code. The Julia implementation is now, in my opinion, fairly mature.

I am asking for comments about replacing our ad-hoc regexp-based parsing mechanisms with it. Specifically, it would help with

1. font locking,
2. indentation,
3. code navigation,

resolving a host of issues.

I am aware that it isn't perfect, as nothing is, but at least improvements would go to a repository that helps all Julia users, not just those who use Emacs.

EDIT Some links:

1. Mastering Emacs has a nice writeup about tree-sitter, and also mentions combobulate which also looks useful
2. tree-sitter docs

[tpapp]

@FelipeLema: since you wrote an indentation solution using tree-sitter, I would appreciate your comments. I recall you mentioning it a while ago.

[FelipeLema]

I agree that currently maintaining the regex parser is taking a toll on code readability and maintanability, even considering people who are well versed in (Emacs) Lisp. I agree that tree-sitter is a tool that has many eyes on them ("it is production quality") and that end users have only good things to say about it.

However, I'd say it gets complicated for Julia. There's 2 big aspects to consider: how well the parser is maintained and how well it can be maintained (by the Julia community).

First, not only is the Julia tree-sitter parser not complete, but its pace may not fit the requirements of the Julia community (incl tooling people & Julia devs). It didn't fit me particularly, I ran into this problem rather quickly. When I started using the tree-sitter indent tool I found a bug. Then I reported it, debugged it, proposed a fix and I'm still waiting on the problem to be addressed. Can the tree-sitter parser be hijacked by the julia community so we can improve it at a quicker pace? I honestly don't know.

After that, I endend up writing a Julia code formatter because I thought it would easier to do so rather than pushing for changes upstream in the Julia tree-sitter parser. The name of this tool I did is actually a misnomer because it parses the code in a Julia buffer and does operations on its AST just as tree-sitter does.

So this brings up the second concern about tree-sitter: it is written in another language (and right now we need to write fixes in JS for the Julia parser). Correct me if I'm wrong, but I believe that most of the Julia community understands the long-term problems of maintaining several languages for their workflows. I would personally rather avoid writing JS if possible (which is what I ended up doing in the paragraph above).

My recommendation is to use CSTParser (or even the parser that comes with Julia binaries) to parse the AST and to copy code from tree-sitter.el to handle the items mentioned at the top entry of this discussion. From my experience with the Julia code formatter I'd recommend using DaemonMode for Emacs-Julia communication (to have little-to-no response delay) as using json-rpc, as LSP does, may bring problems when used in Windows.

Using CSTParser may have a positive effect as it lowers the barriers for Julia end users to participate in maintaining this package (kinda what the Racket community was betting on when they switched to ChezScheme).

All this being said, I want to note that I use tree-sitter in neovim on an everyday basis and that everything (except for the Julia tree-sitter parser) works A-OK.

[tpapp]

@FelipeLema: thanks for the detailed explanation (incidentally, would you consider giving the PR you mentioned a friendly ping? Perhaps it was just overlooked --- that happens).

Conceptually, I can think of the following components for the features we need:

1. a parsing framework, if applicable (eg tree-sitter, LSP)
2. Julia-specific part within that framework (eg tree-sitter-julia, LanguageServer.jl)
3. IDE-specific integration with the framework that calls this code, if applicable (emacs-tree-sitter, emacs-lsp)
4. a bit of mode-specific glue that calls this

Currently in julia-mode we pretty much do everything above with hacks using regexps.

CSTParser.jl does 1+2. We would have to maintain part 3 ourselves, most likely using a daemon-based approach you outline. The advantage is indeed doing a lot in Julia. The disadvantage is all the framework associated with maintaining a running instance of Julia --- doable but somewhat heavyweight.

Tree-sitter would allow us to combine effort for 1, 2, and 3 with other projects (editors other than Emacs, languages other than Julia). That said, I realize that a lot of layers can introduce problems, too.

Interestingly, the LSP spec includes semantic tokens since 3.16. I wonder if that's supported in practice for Julia with Emacs, @gdkrmr and @non-Jedi, it would be great if you could share your thoughts about this. If we could make that work, it would cover pretty much everything for us.

[dahtah]

Note that julia-snail already includes an interface to CSTParser.jl. It still relies on julia-mode for syntax highlighting and formatting, though.

[non-Jedi]

Interestingly, the LSP spec includes semantic tokens since 3.16. I wonder if that's supported in practice for Julia with Emacs, @gdkrmr and @non-Jedi, it would be great if you could share your thoughts about this. If we could make that work, it would cover pretty much everything for us.

LanguageServer.jl doesn't support the semantictokens set of capabilities at this time unfortunately. I'm also really not sure if the architecture of the LSP would give you sufficient responsiveness for indentation and syntax highlighting. Any time you press enter, emacs would have to make a round-trip with the language server (plaintext over pipes) before deciding on indentation level.

[FelipeLema]

worth noting: there's an active (very active? somewhat active?) support for parsing Julia in Scintilla

• original issue asking for lexer support
• PR that solved that issue -> Create julia lexer and tags parser geany/geany#2584
• code has moved to github, this is the latest update on julia -> Improve LexJulia ScintillaOrg/lexilla#13

using Scintilla would have the benefit of an active support, but would have to integrate it to Emacs ourselves

[tpapp]

I did not dig into the details, but I am still under the impression that tree-sitter would be the path of least resistance, because of:

1. existing Emacs integration,
2. existing Julia support, even if we need to fix things,
3. it is fast,
4. no need to spin up a Julia instance, so we could keep things reasonably simple in julia-emacs.

[non-Jedi]

I'm generally on board with integrating with tree-sitter. Even if support for julia syntax isn't perfect, it's probably better than what we have now especially wrt indentation.

We would need to update our lowest supported emacs version to 25 for dynamic module support.

[tpapp]

That's fine with me, Emacs 25 has been released almost 6 years ago.

[gdkrmr]

I don't really have any experience with syntax highlighting in Emacs, please correct me if I am wrong

• I don't think doing it through LanguageServer.jl/CSTParser.jl would be a good idea without a fallback. There is too much that can go wrong if you have to spin up Julia processes in the background. I also like the current modular approach: if for some reason lsp-julia fails, julia-mode is still very much usable (and users can switch to eglot). Including CSTParser.jl would probably mean that we have to put everything in a large monolith (why spin up two separate Julia processes if CSTParser and LanguageServer could run in a single process).
• tree-sitter sounds interesting and I don't mind increasing the minimum Emacs version to 25.
• tree-sitter uses dynamic modules which should be much more responsive than queries to an external Julia binary.

Questions:

• Would it be possible to wrap the Julia Language Server in a dynamic module?
• How does VSCode do syntax highlighting? Are they using CSTParser?

[ronisbr]

Is there anyway to enable tree-sitter in Julia mode? I installed, enabled, but it does not seem to have any effect.

[FelipeLema]

can you paste or point to the code you're dealing with?

[ronisbr]

Nothing in particular. I am just trying to see if we can have a better performance. For example, scrolling this file is somewhat slow for me:

https://github.com/ronisbr/PrettyTables.jl/blob/master/src/backends/text/print.jl