Perl indexing: switch to the more accurate tree-sitter-perl grammar (ts-parser-perl) — ~92% fewer parse failures

[rabbiveesh]

TL;DR

LeanKG's Perl support uses ganezdragon/tree-sitter-perl (the tree-sitter-perl crate, Cargo.toml L56/L78). There's a substantially more accurate Perl grammar — tree-sitter-perl/tree-sitter-perl, published as the ts-parser-perl crate. On real-world Perl it produces ~92% fewer parse failures, which directly translates into more symbols indexed. The catch: because LeanKG matches node-kind names directly (no .scm queries), switching needs a small extractor patch, not just a dependency bump. Details + offer to PR below.

Why it matters for an indexer

Benchmark on 8,342 real-world Perl files (perl5 core + CPAN: DBIx-Class, Mojolicious, SpamAssassin, Bugzilla, …), counting files that parse with no ERROR/MISSING nodes:

grammar	clean parse	failures
ts-parser-perl	95.4%	382
tree-sitter-perl (ganezdragon, current dep)	40.2%	4,983

(On a curated 3,386-module gold corpus: 99.6% vs 73.9%.)

For a parser this is a quality stat; for LeanKG specifically it's a coverage stat. A tree-sitter ERROR node makes that entire subtree untraversable, so on the ~60% of files the current grammar can't parse cleanly, extractor.rs silently walks past subs/packages/calls that live under the error. The better grammar recovers those subtrees.

Why it's not a one-line swap

LeanKG ships no .scm queries — src/indexer/extractor.rs::visit_node matches node kind names directly, and those names were (reasonably) tuned to ganezdragon's grammar, whose Perl node names happen to overlap with the JS/Java/Python grammars (function_definition, method_invocation, import). ts-parser-perl follows the canonical Perl-grammar naming, which doesn't overlap:

concept	ganezdragon (matched today)	ts-parser-perl
sub	function_definition ✅	subroutine_declaration_statement
method def	method	method_declaration_statement
use/import	import ✅	use_statement (module in a module field)
package	package_statement (not in any arm)	package_statement
call	method_invocation ✅	function_call_expression / method_call_expression

So a naive dep swap would make Perl extraction return ~nothing. Two spots need updating:

1. Match arms in visit_node — add subroutine_declaration_statement, method_declaration_statement (functions), use_statement (imports), optionally package_statement (a package/class element), and function_call_expression / method_call_expression (calls). Also add the decl statements to the parent-tracking matches! block lower down.
2. get_node_name — for these nodes the name is in a name field of kind bareword (for use_statement it's a module field; for package_statement/class_statement a name field of kind package). The current generic fallback only looks for identifier-ish children, so it'd return empty without reading those fields.

The dependency change itself can be a Cargo package-rename so the tree_sitter_perl::LANGUAGE path in src/indexer/parser.rs keeps working verbatim:

tree-sitter-perl = { package = "ts-parser-perl", version = "1.1" }

Offer

Happy to send a PR doing both the dep rename and the extractor.rs node-name + field handling. One caveat I'd flag up front: I can verify the grammar's node shapes (from its node-types.json) but can't run a full cargo build/index of LeanKG in my environment, so I'd want CI / a maintainer to confirm the extraction end-to-end. Want me to open it?

(Disclosure: I maintain tree-sitter-perl/tree-sitter-perl. Benchmark is reproducible — compile each grammar to a separate .so and parse the same corpus; methodology in the repo's benchmark/.)