Java is one of the fastest languages to compile (1000 files with ten megs worth of code would take something under 10 seconds), while C++ and Haskell are both very slow to compile even if they're not JVM based.
I suppose we do need to take account of language features. C++ is a difficult language to parse, and with a bit of template magic, and heavy compiler optimizations that can be applied. Haskell has type inference and is statically typed, that has to cost it some what.
Scala has a 8mb JVM library to load, and also type inference. It gets a double whammy. However I'm willing to bet is the JVM is contributing a significant portion of startup time, with both JVM startup time, loading java libraries, then loading scala library. If the JVM wasn't part of the problem, then there is no need for people to create tools like fsc to speed up compile time.
Scala compilation is complex, partly due to its extremely sophisticated type system, it is also quite resource intensive - I have got the type checker to run out of memory on multiple occasions I have never seen a Java compilation fail due to lack of memory. fsc and incremental compilation do speed things up but it is still slower than compiling an equivalent amount of Java.