We only run into big compilation speed problems when we have the following:
1) Code that is lumped into one gigantic JAR instead of being broken down into several different libraries that can be compiled faster.
2) Code that forces the compiler to "solve" many problems in the type system, such as recursion and many other powerful type level constraints.
The first isn't a problem with the compiler, it's a problem with how you're laying out your code. Luckily we don't have this problem.
The second is a problem if you've decided to use a lot of type level programming all over the place. I can't say if this is really a "problem" or not because it depends on your use case.