By switching from the default GNU gold linker to LLVM LLD, I have managed to shave off a few seconds on every build. There's an open issue [0] about switching to LLD. It still has some compatibility issues but it works fine on my machine and albeit tiny project.
I would be curious to see how compile times improve for other people just by linking with LLD. You of course need to install LLD first and then you can tell the chain to use it:
time RUSTFLAGS="-C link-arg=-fuse-ld=lld" cargo build
[0] https://github.com/rust-lang/rust/issues/39915