The internet is full of half-baked "secure" and "private" chat clients and servers. Security is hard to get right. I'm guessing they don't want the reputation of Signal muddled up with some other random client. IIUC anyone can run their own network, and make it as open as they like, using their code, they just can't call it Signal. Much like how Firefox and RedHat use their trademarks.
For all we know, both the Signal network and its client are a half-baked "secure" and "private" chat, and the source code they publish is an elaborate decoy (though probably a subset of the real underlying code, for obvious reasons).
That's a valid question which used to bug me about open source projects. But apparently they finally figured out that the output of the source needs to be deterministic and match the binaries they ship. This property is called "reproducible builds". Signal claims to have them (modulo some third party libraries), though I haven't personally verified it: https://signal.org/blog/reproducible-android/
Honestly though, trust boils down to trusting people. I trust Signal because I trust Moxie, and I trust Moxie because of his reputation among the prominent security experts publicly active on the internet, at least the ones that I find convincing. As a security layman, that's the best I can do.