FWIW cold builds (i.e., in docker with no cache) of cargo are much slower than go, hanging for a long time on refreshing cargo.io indexes. I don't know exactly what that is doing but I have a feeling it is implemented in a monolithic way rather than on-demand. Rust has had plenty of time to make this better but it is still very slow for cold cargo builds, often spending minutes refreshing the crates index. But Go misses easy optimizations like creating strings from a byte slice.
So it is what it is - Go makes explicit promises of fast compile times. Thanks to that, build scripts in go are pretty fast. Any language that doesn't make that explicit might be slow to compile and might run fast - that's totally fine and I would rather have two languages optimized to each case than one mediocre language.