Switch from syn to ra_ap_syntax for parsing

[CathalMullan]

Trying to run cargo shear --expand on the majority of Rust 1.88+ projects will fail due to the use of super let by the pin! macro on nightly, which syn can't parse: rust-lang/rust#139114

syn won't add support for parsing super let until it's closer to stabilization: dtolnay/syn#1947

So I opted to switch to rust-analyzer's syntax parser instead. The API is pretty similar overall. Main benefit is it supports more experimental syntax. Though I'm not sure how stable it is, which might be a concern.

It's a more resilient parser, so it handles syntax errors gracefully. That might backfire in certain scenarios.

With these changes, I was able to run --expand against the Zed monorepo, which found 100+ issues.

Analyzing /home/cmullan/workspace/zed

acp_thread -- crates/acp_thread/Cargo.toml:
  unused dependencies:
    zlog
    tempfile

acp_tools -- crates/acp_tools/Cargo.toml:
  unused dependencies:
    serde

action_log -- crates/action_log/Cargo.toml:
  unused dependencies:
    indoc

agent -- crates/agent/Cargo.toml:
  unused dependencies:
    editor
    terminal
    worktree

agent_servers -- crates/agent_servers/Cargo.toml:
  unused dependencies:
    nix
    libc
    language

agent_settings -- crates/agent_settings/Cargo.toml:
  unused dependencies:
    serde_json_lenient
    serde_json
    paths

ai_onboarding -- crates/ai_onboarding/Cargo.toml:
  unused dependencies:
    serde

assistant_text_thread -- crates/assistant_text_thread/Cargo.toml:
  unused dependencies:
    indoc

audio -- crates/audio/Cargo.toml:
  unused dependencies:
    serde

buffer_diff -- crates/buffer_diff/Cargo.toml:
  unused dependencies:
    serde_json

call -- crates/call/Cargo.toml:
  unused dependencies:
    http_client
    serde

client -- crates/client/Cargo.toml:
  unused dependencies:
    fs

collab -- crates/collab/Cargo.toml:
  unused dependencies:
    context_server
    dap-types
    sea-orm-macros
    agent_settings
    hyper
    audio
    multi_buffer

collab_ui -- crates/collab_ui/Cargo.toml:
  unused dependencies:
    pretty_assertions
    tree-sitter-md

command_palette -- crates/command_palette/Cargo.toml:
  unused dependencies:
    env_logger
    serde_json
    ctor

copilot -- crates/copilot/Cargo.toml:
  unused dependencies:
    async-std
    clock
    client
    task

credentials_provider -- crates/credentials_provider/Cargo.toml:
  unused dependencies:
    serde

dap -- crates/dap/Cargo.toml:
  unused dependencies:
    tree-sitter
    tree-sitter-go

dap_adapters -- crates/dap_adapters/Cargo.toml:
  unused dependencies:
    serde

diagnostics -- crates/diagnostics/Cargo.toml:
  unused dependencies:
    serde
    client
    component

edit_prediction_button -- crates/edit_prediction_button/Cargo.toml:
  unused dependencies:
    theme
    lsp
    serde_json
    futures

edit_prediction_context -- crates/edit_prediction_context/Cargo.toml:
  unused dependencies:
    clap

editor -- crates/editor/Cargo.toml:
  unused dependencies:
    http_client
    tempfile

extension_cli -- crates/extension_cli/Cargo.toml:
  unused dependencies:
    serde

extensions_ui -- crates/extensions_ui/Cargo.toml:
  unused dependencies:
    serde

feedback -- crates/feedback/Cargo.toml:
  unused dependencies:
    editor

file_finder -- crates/file_finder/Cargo.toml:
  unused dependencies:
    language

file_icons -- crates/file_icons/Cargo.toml:
  unused dependencies:
    serde

git -- crates/git/Cargo.toml:
  unused dependencies:
    unindent

git_ui -- crates/git_ui/Cargo.toml:
  unused dependencies:
    windows

go_to_line -- crates/go_to_line/Cargo.toml:
  unused dependencies:
    tree-sitter-typescript
    serde
    tree-sitter-rust

gpui -- crates/gpui/Cargo.toml:
  unused dependencies:
    cocoa-foundation
    x11-clipboard
    pretty_assertions

image_viewer -- crates/image_viewer/Cargo.toml:
  unused dependencies:
    serde

journal -- crates/journal/Cargo.toml:
  unused dependencies:
    serde

json_schema_store -- crates/json_schema_store/Cargo.toml:
  unused dependencies:
    serde

keymap_editor -- crates/keymap_editor/Cargo.toml:
  unused dependencies:
    serde
    component

language_models -- crates/language_models/Cargo.toml:
  unused dependencies:
    project
    editor

languages -- crates/languages/Cargo.toml:
  unused dependencies:
    text
    rope
    workspace

livekit_client -- crates/livekit_client/Cargo.toml:
  unused dependencies:
    sha2
    serde_json
    objc

media -- crates/media/Cargo.toml:
  unused dependencies:
    ctor

multi_buffer -- crates/multi_buffer/Cargo.toml:
  unused dependencies:
    project

net -- crates/net/Cargo.toml:
  unused dependencies:
    anyhow

node_runtime -- crates/node_runtime/Cargo.toml:
  unused dependencies:
    async-std

notifications -- crates/notifications/Cargo.toml:
  unused dependencies:
    settings

outline -- crates/outline/Cargo.toml:
  unused dependencies:
    tree-sitter-typescript

picker -- crates/picker/Cargo.toml:
  unused dependencies:
    env_logger
    ctor

proto -- crates/proto/Cargo.toml:
  unused dependencies:
    typed-path

recent_projects -- crates/recent_projects/Cargo.toml:
  unused dependencies:
    serde
    dap
    task

remote_server -- crates/remote_server/Cargo.toml:
  unused dependencies:
    crash-handler
    git2
    minidumper
    serde
    dap
    workspace

repl -- crates/repl/Cargo.toml:
  unused dependencies:
    env_logger
    serde

reqwest_client -- crates/reqwest_client/Cargo.toml:
  unused dependencies:
    serde
    gpui

rules_library -- crates/rules_library/Cargo.toml:
  unused dependencies:
    serde

schema_generator -- crates/schema_generator/Cargo.toml:
  unused dependencies:
    serde

settings_profile_selector -- crates/settings_profile_selector/Cargo.toml:
  unused dependencies:
    client
    language

settings_ui -- crates/settings_ui/Cargo.toml:
  unused dependencies:
    futures
    zlog
    session
    node_runtime
    client
    assets
    language

supermaven -- crates/supermaven/Cargo.toml:
  unused dependencies:
    project
    theme
    http_client
    env_logger
    editor

tab_switcher -- crates/tab_switcher/Cargo.toml:
  unused dependencies:
    language
    anyhow

tasks_ui -- crates/tasks_ui/Cargo.toml:
  unused dependencies:
    serde

telemetry -- crates/telemetry/Cargo.toml:
  unused dependencies:
    serde

terminal_view -- crates/terminal_view/Cargo.toml:
  unused dependencies:
    rand
    client

text -- crates/text/Cargo.toml:
  unused dependencies:
    http_client

theme_selector -- crates/theme_selector/Cargo.toml:
  unused dependencies:
    serde

title_bar -- crates/title_bar/Cargo.toml:
  unused dependencies:
    pretty_assertions
    tree-sitter-md

ui -- crates/ui/Cargo.toml:
  unused dependencies:
    windows

util -- crates/util/Cargo.toml:
  unused dependencies:
    indoc

vim -- crates/vim/Cargo.toml:
  unused dependencies:
    assets

watch -- crates/watch/Cargo.toml:
  unused dependencies:
    rand

web_search -- crates/web_search/Cargo.toml:
  unused dependencies:
    serde

web_search_providers -- crates/web_search_providers/Cargo.toml:
  unused dependencies:
    serde

workspace -- crates/workspace/Cargo.toml:
  unused dependencies:
    windows
    dap

worktree -- crates/worktree/Cargo.toml:
  unused dependencies:
    serde
    git2

zed -- crates/zed/Cargo.toml:
  unused dependencies:
    profiling
    dap
    task

zeta -- crates/zeta/Cargo.toml:
  unused dependencies:
    reqwest_client
    call
    rpc
    serde
    tree-sitter-go

zeta2 -- crates/zeta2/Cargo.toml:
  unused dependencies:
    lsp

zeta2_tools -- crates/zeta2_tools/Cargo.toml:
  unused dependencies:
    zlog
    indoc
    clap
    serde
    settings
    pretty_assertions

root -- Cargo.toml:
  unused dependencies:
    pet-pixi
    collab
    wit-component
    num-traits
    storybook
    futures-batch
    plugin_macros
    cocoa-foundation
    ai
    hyper
    scheduler
    auto_update_helper
    plugin
    theme_importer
    rich_text


cargo-shear may have detected unused dependencies incorrectly due to its limitations.
They can be ignored by adding the crate name to the package's Cargo.toml:

[package.metadata.cargo-shear]
ignored = ["crate-name"]

or in the workspace Cargo.toml:

[workspace.metadata.cargo-shear]
ignored = ["crate-name"]

To automatically fix issues, run with --fix

[Boshen]

This is freaking nice! I didn't know the existence of ra_ap_syntax when I started the project.

I'm curious about the performance with this parser.

[Boshen]

Thought to myself: do I need to release a v2, this may break some CIs.

Edit: it only breaks with --expand, so major bump is not required.

[Boshen]

It's significantly faster!

oxc  main ❯ hyperfine -i -n before -n after '/Users/boshen/github/cargo-shear/target/release/cargo-shear-before' '/Users/boshen/github/cargo-shear/target/release/cargo-shear'
Benchmark 1: before
Time (mean ± σ): 570.6 ms ± 38.8 ms [User: 1623.1 ms, System: 182.5 ms]
Range (min … max): 539.4 ms … 668.3 ms 10 runs

Warning: Ignoring non-zero exit code.

Benchmark 2: after
Time (mean ± σ): 425.8 ms ± 12.3 ms [User: 1118.4 ms, System: 185.3 ms]
Range (min … max): 406.5 ms … 442.4 ms 10 runs

Warning: Ignoring non-zero exit code.

Summary
after ran
1.34 ± 0.10 times faster than before

[lmmx]

Great to see this!

I expect one optimisation that could be made is to avoid .to_string()ing entire syntax nodes

cargo-shear/src/import_collector.rs

Lines 163 to 167 in ab9e9a7

	fn collect_tokens(&mut self, node: &SyntaxNode) {
	let source_text = node.text().to_string();
	let idents = source_text
	.match_indices("::")

Instead by walking the tokens

fn collect_tokens(&mut self, node: &SyntaxNode) {
    let mut prev_token = None;
    for token in node.descendants_with_tokens().filter_map(|e| e.into_token()) {
        if token.text() == "::" {
            if let Some(prev) = prev_token {
                if let Some(ident) = Self::extract_identifier_before(prev.text(), prev.text().len()) {
                    self.add_import(&ident);
                }
            }
        }
        prev_token = Some(token);
    }
}

but better still to use the token kind to do the same check but type safe

fn collect_tokens(&mut self, node: &SyntaxNode) {
    for elem in node.descendants_with_tokens() {
        if let Some(token) = elem.as_token() {
            if token.kind() == SyntaxKind::COLON2 {
                // Look at previous token
                if let Some(prev) = token.prev_token() {
                    if let Some(ident) = Self::extract_identifier_from_token(&prev) {
                        self.add_import(&ident);
                    }
                }
            }
        }
    }
}

Away from desk otherwise would run checks with these suggestions, sorry (: