This is why there is the "Reproducible Builds"[0] initiative from Debian folks.

[0] https://reproducible-builds.org/

That's true and, to me, has been brought on the spotlight from Microsoft practices around VSCode. On the other hand, for an OSS project, building the binary is not that hard.