Skip to content

Commit

Permalink
Add rust-analyzer proc-macro server project
Browse files Browse the repository at this point in the history
  • Loading branch information
Veykril committed Feb 22, 2025
1 parent eab1c6c commit e5de328
Showing 1 changed file with 49 additions and 6 deletions.
55 changes: 49 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ We use the GSoC project size parameters for estimating the expected time complex
- [Prototype Cargo plumbing commands](#prototype-cargo-plumbing-commands)
- [Move cargo shell completions to Rust](#move-cargo-shell-completions-to-Rust)
- [Build script delegation](#build-script-delegation)
- **rust-analyzer**
- [Implement a new proc-macro server RPC API](#proc-macro-rpc)
- **Crate ecosystem**
- [Modernize the libc crate](#Modernize-the-libc-crate)
- [Add more lints to `cargo-semver-checks`](#add-more-lints-to-cargo-semver-checks)
Expand Down Expand Up @@ -257,7 +259,7 @@ Medium.

**Description**

Over the last year, support for automatic differentiation ('autodiff') was added to the Rust compiler. The autodiff tool which we are using ([Enzyme](https://enzyme.mit.edu/)) operates
Over the last year, support for automatic differentiation ('autodiff') was added to the Rust compiler. The autodiff tool which we are using ([Enzyme](https://enzyme.mit.edu/)) operates
on LLVM-IR, which is the intermediate representation of code, used by LLVM. LLVM is the default backend of the Rust compiler. Unfortunately, two layout related problems limit its usability.

A) The Rust compiler has a set of ABI optimizations which can improve performance, but make it harder for autodiff to work. An example is the function `fn foo(a: f32, b: f32) -> f32`,
Expand All @@ -267,7 +269,7 @@ If a function has a `#[rustc_autodiff]` attribute, the Rust compiler should simp
Multiple examples of function headers which will get handled incorrectly at the moment are listed [here](https://github.com/EnzymeAD/rust/issues/105).

B) Enzyme requires good information about the memory layout of types, both to be able to differentiate the code, and to do so efficiently. In order to help Enzyme,
we want to lower more Type Information from MIR or even THIR into LLVM-IR metadata, or make better usage of existing debug info. If you are interested in this part and
we want to lower more Type Information from MIR or even THIR into LLVM-IR metadata, or make better usage of existing debug info. If you are interested in this part and
also have some LLVM experience, please have a look at the LLVM website for the related proposal.

For both A) and B), the online compiler explorer [here](https://enzyme.mit.edu/explorer/) can be used to trigger both types of bugs, to get a feeling for existing problems.
Expand Down Expand Up @@ -310,11 +312,11 @@ Currently, the backend end (codegen part) of the compiler has been parallelized,
The most important and valuable work in this area are two aspects:

A) Diagnosing and fixing deadlock [issues](https://github.com/rust-lang/rust/issues?q=is%3Aopen+label%3AWG-compiler-parallel+deadlock) caused by the execution order of compiler queries in a multithreaded environment.
[Queries](https://rustc-dev-guide.rust-lang.org/query.html) is a unique design of the Rust compiler, which is used to achieve incremental compilation process. It divides the compiler
process into various parts and caches the execution results of each part. However, queries caching dependencies between multiple threads may cause deadlock.
[Queries](https://rustc-dev-guide.rust-lang.org/query.html) is a unique design of the Rust compiler, which is used to achieve incremental compilation process. It divides the compiler
process into various parts and caches the execution results of each part. However, queries caching dependencies between multiple threads may cause deadlock.
[`Work-stealing`](https://en.wikipedia.org/wiki/Work_stealing), a method used to improve parallelization performance, is the core reason.

To solve these problems, we need to find the part of the compiler process that causes deadlock through diagnosing coredumps in issues, and adjusting the execution order
To solve these problems, we need to find the part of the compiler process that causes deadlock through diagnosing coredumps in issues, and adjusting the execution order
of this part of code so that there will be no circular dependencies on the query caches between multiple threads. This [PR](https://github.com/rust-lang/rust/pull/118488) is a good example of solving a deadlock problem.

B) Improving the performance of the parallel frontend
Expand Down Expand Up @@ -611,6 +613,47 @@ Medium.
- [Idea discussion](https://rust-lang.zulipchat.com/#narrow/channel/421156-gsoc/topic/Idea.3A.20Build.20script.20delegation)
- Ed Page ([GitHub](https://github.com/epage), [Zulip](https://rust-lang.zulipchat.com/#narrow/dm/424212-Ed-Page))

## rust-analyzer

### Implement a new proc-macro server RPC API

**Description**

Today, rust-analyzer (and RustRover) is expand proc-macros by spawning a separate proc-macro server
process that loads and executes the proc-macro dynamic libraries. They communicate to this process
via a JSON RPC interface that has not been given much thought when it was implemented, now starting
to show its limitations.

The goal is to replace this current implementation entirely in favor of a more performant format
that also supports the more complicated needs of the proc-macro API.

**Expected result**

1. Replace the JSON format with [postcard](https://docs.rs/postcard/latest/postcard/)
2. Replace the current client to server request-response RPC API with one that allows bidirectional
request-response schemes.
3. Implement multiplexing connections between a client and server (to potentially enable parallel
proc-macro expansion).

**Desirable skills**

Intermediate knowledge of Rust.

**Project size**

Medium.

**Difficulty**

Medium.

**Mentor**
- Lukas Wirth ([GitHub](https://github.com/veykril), [Zulip](https://rust-lang.zulipchat.com/#narrow/dm/300586-Lukas-Wirth))

**Zulip streams**
- [Idea discussion](https://rust-lang.zulipchat.com/#narrow/channel/185405-t-compiler.2Frust-analyzer/topic/proc-macro.20server.20IPC.20format)
- [rust-analyzer team](https://rust-lang.zulipchat.com/#narrow/channel/185405-t-compiler.2Frust-analyzer)

## Crate ecosystem

### Modernize the libc crate
Expand Down Expand Up @@ -833,5 +876,5 @@ Some of the work is medium. Diagnosing and / or fixing failures is often pretty

**Further resources**

- [Wild linker](https://github.com/davidlattimore/wild)
- [Wild linker](https://github.com/davidlattimore/wild)
- [Blog posts, most of which are about Wild](https://davidlattimore.github.io/)

0 comments on commit e5de328

Please sign in to comment.