Strict Rust highlighting is not a good fit inside macro invocations

[jordanhalase]

Reproduction steps

1. Start Zed with Rust syntax highlighting, Semantic Tokens OFF.
2. Type in a valid arbitrary custom macro invocation: e.g., foo! { bar- <b"none"> }
3. Tokens are not highlighted correctly (particularly the string literal)

Current vs. Expected behavior

Currently Zed treats syntax highlighting within a Rust macro as Rust code with hard-coded exceptions for view!, html! and now bsn!. These hard-coded exceptions are good for special cases, but for all other macros, syntax should be highlighted generically that won't break whatever syntax a programmer may come up with in a macro.

In a general sense, it is okay to highlight the innards of a macro invocation loosely as Rust, such for key words like let and while, known literals and identifiers, but treating it strictly as Rust leads to inconsistent highlighting issues. These can be fixed as follows:

1. Keep all PUNCT tokens (@!#.,<>?/ etc.) the same color/style (< does not need a corresponding > in macros and may be grouped with any other PUNCT, e.g., Brainfuck)
   a. ()[]{} can remain styled differently because they are required to enclose each other.
   b. A literal immediately preceding ! can remain a unique color to look like an inner_macro! invocation
   c. A literal immediately preceding ( can remain a unique color to look like a function call
2. Allow PUNCT tokens directly next to string literals without breaking the highlighting: e.g., <c"cstring", <"foo"
3. b and c do NOT signify c-string/bytestrings if part of a literal, i.e., 123c"foo" tokenizes as 123c followed by "foo"
   (Zed currently highlights 123c"foo" incorrectly, with colors separated at 123 and c"foo")

If the above points are fixed, then Rust code written inside of macros will continue to look like Rust code highlighting, while also allowing developers to create any arbitrary custom macros without immediately breaking the highlighting.

Current

Expected

(VS Code handles this perfectly)

Zed version and system specs

Zed 0.227.1
9f9b707

0.227.1+stable.199.9f9b707d39d62468d4cdaed2d8eb95a847da4cf9

Windows 11/macOS Tahoe 26.3.1

[ChristopherBiscardi]

Thank you for following up on filing this! The description is great.

[jordanhalase]

This is also how VS Code handles literals before :: inside macros. I think this is worth preserving for Rusty code in macros that fills in structs, references other modules, etc. (This is how this issue was first discovered in bevy+Zed)

I think it's overall a good idea to have all (non-number) literals be the same color as literals preceding a single : to reduce logic needed. That way you get the "before-:" highlighting for free.

Literals immediately preceding :: turn green, but if there's a space before the :: it stays blue.

[Dnreikronos]

I will invest some time to understand some way to solve that and try to submit something

[Dnreikronos]

I traced the root cause. Zed re-parses everything inside Rust macro invocations as standalone Rust code (lines 7-15 in injections.scm). When a macro uses syntax that isn't valid Rust, like bar- <b"none">, the parser creates ERROR nodes and highlighting breaks. Strings disappear, punctuation gets wrong colors.

PR #51353 worked around this for bsn! by adding it to an exclusion list next to view! and html!. That's whack-a-mole. Every macro with custom syntax needs its own exception.

Tree-sitter-rust already parses token_tree children correctly without help. Identifiers, string literals, numbers, keywords, punctuation, all proper nodes. The existing highlights.scm rules match them. The self-injection is what breaks things.

What's been tried before:

• Punctuation next to string literal breaks syntax highlighting #51240 reported @"foo" and :"bar" breakage (closed via the bsn workaround)
• languages: Prevent bsn macro from injecting rust layer #51353 added bsn to the exclusion list — bandaid
• tree-sitter/tree-sitter-rustMiscellaneous bug fixes after switching to a project-centric sharing model #303 tried making the grammar accept more punctuation prefixes before strings. Wrong direction, closed.

Plan: all 3 points, no highlighting regression:

Instead of re-parsing macro contents as Rust (which chokes on non-Rust syntax), I want to add tree-sitter query patterns that match common Rust constructs directly inside token_tree nodes. We get correct highlighting for arbitrary syntax and keep most of the semantic richness.

Step 1: Kill the self-injection (fixes points 1 and 2):
Delete lines 7-15 in crates/languages/src/rust/injections.scm.

Step 2 — Add token_tree patterns to highlights.scm:

- (token_tree (identifier) @function . (token_tree)) — identifier before (...) = function call
- (token_tree "." . (identifier) @function.method . (token_tree)) — .method(...)
- (token_tree (identifier) @function.special . "!") — nested macro invocations

Stuff that already works without changes: keywords (let, if, fn), uppercase identifiers as types, ALL_CAPS as constants, all literal types.

We keep function calls, method calls, macro invocations, types, constants, keywords, and literals highlighted. We lose standalone field access (x.field without ()) and scoped paths (Foo::bar::Baz), both uncommon inside macros. Matches what VS Code does. Semantic Tokens via rust-analyzer fills in the rest if anyone wants it.

Point 3 (123c"foo") is upstream in tree-sitter-rust. The regex /[bc]?"/ in grammar.js treats c" or b" as a string prefix regardless of what came before. Needs an upstream PR moving prefix detection into the external scanner where preceding context is available. Different from the closed #303, which tried adding more prefix characters instead of restricting existing ones.

What you guys think, @jordanhalase and @ChristopherBiscardi?

[jordanhalase]

Instead of re-parsing macro contents as Rust (which chokes on non-Rust syntax), I want to add tree-sitter query patterns that match common Rust constructs directly inside token_tree nodes. We get correct highlighting for arbitrary syntax and keep most of the semantic richness.

We keep function calls, method calls, macro invocations, types, constants, keywords, and literals highlighted. We lose standalone field access (x.field without ()) and scoped paths (Foo::bar::Baz), both uncommon inside macros. Matches what VS Code does. Semantic Tokens via rust-analyzer fills in the rest if anyone wants it.

Point 3 (123c"foo") is upstream in tree-sitter-rust. The regex /[bc]?"/ in grammar.js treats c" or b" as a string prefix regardless of what came before. Needs an upstream PR moving prefix detection into the external scanner where preceding context is available.

I think this is the perfect approach @Dnreikronos. I think re-parsing macro invocations as strict Rust is a mistake, and highlighting the least common denominator between Rust and any arbitrarily thought-up Rust superset that can fit in a TokenStream is the right approach like you explained here.

(VS Code does special highlighting in macros for literals immediately preceding a :: to look like scoped paths, but they also use the same color for literals whether or not they're behind a single : (see my comment above), but that may not be as important as just fixing the highlighting first.)

Most importantly are getting all macros highlighted as not strict Rust to fix strings and punctuation, and fix the b and c parsing on string literals.