Guideline for wrapping REST server in idiomatic Rust API

[dtolnay]

Let's recommend an idiomatic way to structure a Rust crate that provides an interface for an external JSON server (or similar), such as wdsapi for the IBM Watson Discovery Service or github-rs for GitHub's API. Lots of crates already do this and each one has had to reinvent how to design the top-level API, how to expose the error when a request fails, how to support asynchronous requests, etc.

I think this may work well as part of a broader guideline that recommends crates to look to for guidance on structuring crates for similar tasks. There is some overlap with the objectives of the Rust Cookbook, but that one is more for transferable tasks (unzip an archive, generate a random number) and this would be for cases where the entire structure of the crate is transferable (client for a REST API, FFI bindings for a native library).

@bruceadams -- thanks for the suggestions!

[KodrAus]

Sorry I've been meaning to circle back here for a while and leave my thoughts. This is mostly just an abstract braindump. I'm really interested to hear what others think in this space and work towards some concrete recommendations!

For high-level APIs I'm not sure what we could recommend in the general sense. If the API you're providing a client for has a proper hypermedia focus like GitHub then the methods you expose to build requests might make heavy use of composable builders (I think github-rs does an excellent job of this). Whereas if you're exposing an API that doesn't have a lot of resources then you might not need a sophisticated high-level API.

Async

I think supporting async through tokio is an interesting question. In reqwest we effectively have 2 clients; a core async one and a sync one that wraps the async client and bends over backwards to make sure you don't end up deadlocking on the core thread.

Just offering an async client and providing a synchronous shim that calls .wait can be dangerous with tokio right now because of the potential to deadlock body streams. For this reason I would recommend building REST API clients on reqwest rather than directly on hyper, but that means you need totally separate sync and async client types.

I've previously offered a single client and used a generic parameter to offer different implementations of certain functions depending on whether it's sync or async. I wouldn't recommend this approach. It means some code can be shared if that client is threaded through a lot of request builders that have a lot of methods, but it's very complicated, and makes the crate docs much harder to follow.

I would prefer to just be able to use a thin synchronous shim (or an async-only client) and let the fact that your sync code won't be running on the proposed default tokio core thread reduce the risk of accidentally deadlocking, so it's safe to .wait. In the meantime, my ideal client might look something like:

• Core async client that takes simple datastructures for requests using reqwest's AsyncClient
• Core sync client that takes the same simple datastructures for requests using reqwest's SyncClient
• High-level builders that aren't coupled to a particular client, so you construct a request and then pass it to either client to send. What these look like exactly isn't really important. It'll depend on the domain you're likely to use the client in

If we could just expect users to .wait requests if they need them to behave synchronously then maybe:

• Core async client that takes simple datastructures for requests using reqwest's AsyncClient
• High-level builders could keep a reference to the async client so you can ergonomically build up a request and call .send or .execute or what have you at the end, and then just .wait that if you want it to behave synchronously

Error management

The approach I've started taking is to have two classes of error:

• Domain-specific errors that could be reasonably handled programmatically, like a missing resource you might want to create if it's missing or just ignore
• Domain-specific errors that couldn't be reasonably handled programmatically, like a malformed query
• Transient/transport errors

In the first case, having an ergonomic way to interrogate the error to know what happened is important. In the other cases, having enough detail that can be logged and used to diagnose issues is important. I feel like a black-box with methods like is_missing is better than enums because you have more control to add new variants without breaking. Even if you have a non-exhaustive variant with an enum you need to be careful not to change the conditions under which a variant is returned, or you'll still potentially break consumers.