In unoptimized builds often the linker is the bottleneck. Rust/Cargo can parallelize most of the build, generating tons of code and debug info, but then the poor linker has to consume all of it at once. The object/exe formats were designed in ancient times, so they're hard to build incrementally or in parallel (some linkers are trying).
At least for the fully static binary part of rust, there should be some optimizations there w.r.t. compilation. Sure you're not going to interface with shared libraries well but maybe a small experimental feature for fully owned projects? Idk.
ABIs won't help automatically. The overhead comes mainly from generics and macros. Cross-crate monomorphic generics and AST macros are impossible to express in an ABI. Turning them into polymorphic usually wouldn't remove the many abstraction layers that need to be peeled off, and only make them cost at runtime too.
So far MIR optimizations were helpful. Rust can simplify high-level MIR quicker than LLVM can simplify the same code after it's been lowered to lots of low-level instructions.