Good question. For one, there's also a custom optimized-debug backend that's used by default for Debug builds which is a lot faster than LLVM. Secondly, there's incremental compilation with an object cache so minor changes only general minor rebuilds.
For a clean release build, however, LLVM will dominate the compile time as expected. Compared to C/C++, however, there is less duplication of ODR'd code (ie- methods defined in headers such as template methods) and less debug information duplication so backend time is lower.