You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An important observation that anyone familiar with popular Tree-sitter grammars may have picked up on is that there are no nodes representing things like functions, macros, types, and other semantic concepts.
114
-
Representing the semantics of Clojure in a Tree-sitter grammar is much more difficult than traditional languages that do not use macros heavily like Clojure and other lisps.
115
-
To understand what an expression represents in Clojure source code requires macro-expansion of the source code.
116
-
Macro-expansion requires a runtime, and Tree-sitter does not have access to a Clojure runtime and will never have access to a Clojure runtime.
117
-
Additionally Tree-sitter never looks back on what it has parsed, only forward, considering what is directly ahead of it. So even if it could identify a macro like `myspecialdef` it would forget about it as soon as it moved passed the declaring `defmacro` node.
118
-
Another way to think about this: Tree-sitter is designed to be fast and good-enough for tooling to implement syntax highlighting, indentation, and other editing conveniences. It is not meant for interpreting and execution.
115
+
An important observation that anyone familiar with popular Tree-sitter grammars
116
+
may have picked up on is that there are no nodes representing things like
117
+
functions, macros, types, and other semantic concepts. Representing the
118
+
semantics of Clojure in a Tree-sitter grammar is much more difficult than
119
+
traditional languages that do not use macros heavily like Clojure and other
120
+
Lisps.
121
+
122
+
To understand what an expression represents in Clojure source code
123
+
requires macro-expansion of the source code. Macro-expansion requires a
124
+
runtime, and Tree-sitter does not have access to a Clojure runtime and will
125
+
never have access to a Clojure runtime. Additionally Tree-sitter never looks
126
+
back on what it has parsed, only forward, considering what is directly ahead of
127
+
it. So even if it could identify a macro like `myspecialdef` it would forget
128
+
about it as soon as it moved passed the declaring `defmacro` node. Another way
129
+
to think about this: Tree-sitter is designed to be fast and good-enough for
130
+
tooling to implement syntax highlighting, indentation, and other editing
131
+
conveniences. _It is not meant for interpreting and execution._
119
132
120
133
#### Example 1: False Negative Function Classification
121
134
@@ -128,8 +141,11 @@ Consider the following macro
128
141
(defn2dog [] "bark")
129
142
```
130
143
131
-
This macro lets the caller define a function, but a hypothetical tree-sitter-clojure semantic grammar might just see a function call where a variable dog is passed as an argument.
132
-
How should Tree-sitter know that `dog` should be highlighted like function? It would have to evaluate the `defn2` macro to understand that.
144
+
This macro lets the caller define a function, but a hypothetical
145
+
tree-sitter-clojure semantic grammar might just see a function call where a
146
+
variable dog is passed as an argument. How should Tree-sitter know that `dog`
147
+
should be highlighted like function? It would have to evaluate the `defn2` macro
148
+
to understand that.
133
149
134
150
#### Example 2: False Positive Function Classification
135
151
@@ -154,13 +170,17 @@ How is Tree-sitter supposed to understand that `(defn foo [] 2)` of the expressi
154
170
155
171
#### Syntax and Semantics: Conclusions
156
172
157
-
While these examples are silly, they illustrate the issue with encoding semantics into the tree-sitter-clojure grammar.
158
-
If we tried to make the grammar understand functions, macros, types, and other semantic elements it will end up giving false positives and negatives in the parse tree.
159
-
While this is an inevitability for simple static analysis of Clojure code, tree-sitter-clojure chooses to avoid making these kinds of mistakes all-together.
160
-
Instead, it is up to the emacs-lisp code and other consumers of the tree-sitter-clojure grammar to make decisions about the semantic meaning of clojure-code.
161
-
162
-
There are some pros and cons of this decision for tree-sitter-clojure to only consider syntax and not semantics.
163
-
Some of the (non-exhaustive) upsides:
173
+
While these examples are silly, they illustrate the issue with encoding
174
+
semantics into the tree-sitter-clojure grammar. If we tried to make the grammar
175
+
understand functions, macros, types, and other semantic elements it will end up
176
+
giving false positives and negatives in the parse tree. While this is an
177
+
inevitability for simple static analysis of Clojure code, tree-sitter-clojure
178
+
chooses to avoid making these kinds of mistakes all-together. Instead, it is up
179
+
to the emacs-lisp code and other consumers of the tree-sitter-clojure grammar to
180
+
make decisions about the semantic meaning of clojure-code.
181
+
182
+
There are some pros and cons of this decision for tree-sitter-clojure to only
183
+
consider syntax and not semantics. Some of the (non-exhaustive) upsides:
164
184
165
185
- No semantic false positives or negatives in the parse tree.
166
186
- Simple grammar to maintain with less nodes and rules
0 commit comments