Don't trust Moxie; don't trust the server side of Signal.
Which leads to one of the selling points of Signal: that the client is designed to expose as little metadata as possible to the server. Sure, the Signal servers could be slurping up IP addresses and timings despite advertising that they don’t. But I can’t think of a single alternative service that can guarantee otherwise, and that problem is compounded by clients that leak much more metadata than Signal.
This comes across as dismissive of the criticisms outlined by GP in favor of more general critiques about owning your own data. Yes, "The cloud is just someone else's computer", but GP gave specific criticisms about Signal.
This would be no different from someone replying to a criticism about Telegram ("They rolled their own crypto; Don't trust the server side of Telegram!") with "Don't trust the server side of any service.". Yes, it's true, but it doesn't address the issue raised.
The difference compared to your analogy is that Telegram’s self‐rolled crypto is a problem many messengers don’t have, and it can’t be mitigated by the client.
Whereas the possibility of malicious things being done on Signal’s server end is a problem shared by every messenger, and Signal works hard to mitigate this by pushing smarts like contact handling entirely to the client side.
I also don’t see why you brought up “owning your own data.” I already discussed that the same “don’t trust external servers” axiom applies to self‐hosted federated systems as well.
I'm pretty sure I can check what code my XMPP server in my room runs. Not a freedom you have when using something like WhatsApp or Signal.
Besides, it's their service, they get to say who and how they connect.