App being open-source means you can compile and build it yourself.
They've also gone to extreme lengths to prove that the software running on their servers has not been tampered with: https://signal.org/blog/private-contact-discovery/#trust-but...
I think your doubts create very unreasonable expectations for Signal.