That's a valid question which used to bug me about open source projects. But apparently they finally figured out that the output of the source needs to be deterministic and match the binaries they ship. This property is called "reproducible builds". Signal claims to have them (modulo some third party libraries), though I haven't personally verified it: https://signal.org/blog/reproducible-android/

Honestly though, trust boils down to trusting people. I trust Signal because I trust Moxie, and I trust Moxie because of his reputation among the prominent security experts publicly active on the internet, at least the ones that I find convincing. As a security layman, that's the best I can do.