Is it mostly compilation time, or link time? At least for debug builds, I found swapping out the linker to work wonders. There are some examples on https://endler.dev/2020/rust-compile-times/ (no affiliation, just 1st google result)
It enables dead code elimination, and removes a lot of code very early.