It's not even on f-droid. You install the binary they control and connect to the servers they run, it's literally a proprietary app, just like Telegram.
They've also gone to extreme lengths to prove that the software running on their servers has not been tampered with: https://signal.org/blog/private-contact-discovery/#trust-but...
I think your doubts create very unreasonable expectations for Signal.
It doesn't get much more open source than that.