Better use a purely functional build system, like in NixOS, where you can download binaries and have the guarantee that they are the same as when you had built them yourself.
It's much closer to that than any alternative I am aware of, but that's one guarantee you don't have. It's important to limit your substituters (binary caches) to trusted servers, because Nix does not and cannot verify that the binary it receives corresponds to the source, only that the remote server claims it does. Even if the binary does correspond to the source, the build process may have picked up small details about the build system, meaning a local build would produce a different binary. Even if the build process does not pick up any such details, many packages do not have deterministic builds, meaning they might produce different binaries even if you build them multiple times in succession on your own computer.
This is true. I am a NixOS fan, but when a piece of software asks me to use a Nix binary cache it can mask problems like a private git repo or other issues that would prevent a full build without me knowing. Worse yet, a Hydra box eating it's own dogfood ...
Of course you can also use Nix or Guix in a more Gentoo-like fashion and forbid binary downloads, requiring everything to be built locally.