Sort of. The Rust compiler has gotten much faster in the last few years. On most common crates it's become 25-30% faster (https://arewefastyet.pages.dev). This page isn't up to date with the latest releases, which have also become faster.
During development the most typical configuration would be debug-incremental builds. Those are pretty damn fast. And personally, I don't even use rustc during development to figure out errors. rust-analyzer does a great job of giving all errors in line. I think most others who use one of the IDE options might not be waiting on rustc much.
Honestly, "rust is slow" is a bit of a meme at this point. There are engineers working on triaging compiler performance regressions every week, as well as engineers working on improving performance. But there's no concerted effort to communicate these efforts to the community. Which leads to this meme being perpetuated. I guarantee that even if the compiler improves by a further 50%, there will still be comments saying it's too slow, simply because commenters aren't aware of changes over time.
But it _is_ slow. Same as C++ and Haskell compilers are slow. And you cannot get it to really compile faster - like OCaml speed - because the language (macros and traits I guess are the worst) prohibits that. Of course, if all somebody knows are C++, Haskell and Rust the compilation doesn't seem slow to them.
This is a common assumption but I've rarely see cargo check, which both expands macros and solves traits as well as runs the borrow checker take more than a few seconds on my machine, even when full compilation takes minutes. Most of the slowness is really in the parts that come afterwards, I suspect especially once you hit the generics explosion in later passes.
Microsoft packages a full set of binary libraries with a build.rs script for Rust/WinRT, otherwise no one would bother to wait for the build.
https://github.com/microsoft/windows-rs/tree/master/.windows
Also seeing the same crate consumed in multiple version being compiled multiple times isn't fun.
For example, the image crate.
So while most of the compile time is indeed usually in LLVM, there are still a lot of things rustc can do to give LLVM an easier time.