The big problem for me is the compilation time, which is very slow (this may or may not be because the compiler itself is written in Rust).
I dedicated a few days to hacking on rustc this summer, and while the tooling and community are very nice, the compilation times are really high. Git pull && make ? 20 min. Edit a file && make ? 5 min. IIRC the major was that metadata got serialized/unserialized all the time, but anyway it's one of the priorities. I'll hack again on it with pleasure when I'll have more time.