Go would have compiled an order of magnitude faster still. The compiler is less safe, and thorough than Rust's, but still.
How hard is the caching to set up, especially in a CI setting?
Cargo caches by default and you can specify the path that it stores artifacts. Most of the cache story is about what your chosen provider uses to specify build steps, etc