New proc-macro server RPC API

[Veykril]

The current proc-macro server RPC API is reaching its limits preventing us from implementing more of the proc-macro API.

The current API is effectively a serial JSON based client to server request-response via stdout. That is the client (rust-analyzer, RustRover) makes a request to the proc-macro server process via JSON message and then receives a response from the server. Notably these requests cannot interleave (the proc-macro server is sequential).

This has a couple of downsides:

• JSON: We have a weird serialization scheme for JSON to bring down the overall data we sent over the wire to combat JSON being fairly bulky. We want to replace this part (presumably with postcard)
• client to server request-response scheme: A fairly simple messaging style that unfortunately does not suffice for implementing all the proc-macro APIs. Some of the proc-macro functions actually need to access semantic state which lives in the client and hence the server needs to be able to ask the client for this information while it is computing the response for a client's request. That means, we need a request-response scheme from client to server that allows to have server to client request-response pairs while the client is waiting for the response. One can think of this as a client request opening a communication channel from the server to the client, and the server response closing it.
• sequential server: The current implementation of the server is entirely sequential, it can only serve a single request at any given time. Ideally we would allow the server to serve multiple requests concurrently (that is multiplex the connection), but that alone does not parallelize that well due to proc-macros being able to observe its environment and therefore we can only parallelize proc-macro expansions that share the same environment (which depending on the workload is still a win). An alternative would be to allow for spawning multiple proc-macro servers to interface with (this would more require work in the client than the server).

Notably an implementation of this needs to live side by side with the old protocol for a transition period as the proc-macro server is a rustup component that IDEs rely on.

Relevant crates:

• proc-macro-srv is the actual implementation of the proc-macro server logic, irrespective of the RPC API. This will need the least changes.
• [proc-macro-api]https://github.com/rust-lang/rust-analyzer/tree/master/crates/proc-macro-api is the definition of the RPC protocol.
• proc-macro-srv-cli is the entry point of the server. It is responsible for message handling and talking to the client etc.

[davidbarsky]

For posterity, we had some discussion on Zulip as well.

[Shourya742]

Hello @Veykril, I had a couple of questions:

1. Why are we going with Postcard instead of Bincode?? Bincode has a simpler encoding and is quite fast.... The main advantage I see for Postcard is better compression, but we already have the flat.rs module, which converts token trees into a specific flat array structure to avoid stack overflows with JSON. Since we already have this flattened representation, compression doesn’t seem like a necessity. Given that performance is a key concern, wouldn’t Bincode be a better fit?
2. Should the new design support bidirectional streaming communication, or will it stick to a request-response model with nested requests?
   Let me know what you think.....

[Veykril]

I believe the thread linked by David should roughly answer those questions, please read through that. If it doesn't let me know and I'll clarify next week.

[Shourya742]

I checked the attached link, but it redirected me to your comment on finalization. Oh, I see—it’s the thread above that. Thanks! I’ll review it and let you know if I have any question

screen shot