The current slowness of the Rust compiler is largely due to putting more developer attention into improving the runtime speed of fully-optimized binaries. Hence, regardless of which mode it's in, the compiler does a lot of unnecessary work that a debug-mode binary doesn't need or benefit from. There's also simply a ton of low-hanging fruit lying around: we're only two weeks out from 1.0 and the nightly compiler is already reportedly 30% faster for code compiling in the wild. There's another 10% compilation speed reduction coming from work around optimizing linking. There's also work towards parallel codegen coming along nicely (https://github.com/rust-lang/rust/pull/26018) which reports a 30% reduction in compilation time. In the longer run there are plans for incremental compilation which should drastically reduce the amount of work that gets unnecessarily repeated with each compilation cycle (https://github.com/rust-lang/rfcs/pull/594). In the even longer run there are plans to fully parallelize the compiler phases and also to pre-optimize code before it gets to LLVM in order to reduce the sheer quantity of IR that we shovel into it.

TL;DR: Crystal manages much faster compilation than Rust because Rust's compiler developers may have deprioritized optimizing the compiler for a bit too long. :)