The memory safety checking (or more generally, the type checking) isn't what makes the compiler slow. That part is pretty fast because it's intentionally modular.

It's the cross-module optimizations that make the compiler slow. Those optimizations are in service of C++-inspired ideal of being able to use layers of abstraction to make your code cleaner, but having the compiler flatten them down so your program still runs fast.