I think a big issue is that even if it takes a few more seconds, it is still a big win compared with C and C++ builds, usually measured in hours.
But I guess it is a block for those used to programming languages with traditional interpreter implementations.
My biggest complain when trying out Rust was the C++ build times of the bootstrapping process. As for using pre-compiled Rust, I think the compile times are pretty acceptable for 1.0.