Compiling Servo, a web browser engine, takes 5-6 minutes for an optimized build from scratch on my 6C/12T machine. So no, 2 hour build times are not normal for all but the largest software projects.
Compiling paru, an AUR helper for Arch Linux, on the other hand, takes me around 3 - 4 minutes on a 4C/8T i5 for an optimised build. I think Rust compile times might just fall within a narrow range.