Do something about the big rust build overhead

[tlater-famedly]

We have a lot of cargo extensions that are installed in this docker image. Building them takes 20-30 minutes, this is one of our longest (and probably most expensive) CI builds.

We should consider using cargo-binstall to escape this, since pre-built binaries are available for practically all our inputs. Security implications exist, but there are probably easier ways to break our supply chain.

Failing that, we should probably at least consider using the faster builders for some workflows.