Indeed, that's precisely why it's such an important protection mechanism. But it's about more than just not having to trust the original author's compiler. The original author might maliciously copy-paste some additional source code into the build process just before compilation.
Deterministic builds are a way for any of us to download any source code, build it, and verify that all of us are using a binary derived from exactly that source code, and nothing else.