If your builder is compromised, it can be co-opted to sign and verify the "source code" files with any values. The risk of placing this trust in the builder or the nix store is an easy one to avoid. Getting the authencity of the code from the source code independently ought to be the correct way of verifying reproducible builds.