Don’t trust the server side of any service. Open source or not, federated or not, it’s impossible to verify that a server you talk to runs the code it says it does.

Which leads to one of the selling points of Signal: that the client is designed to expose as little metadata as possible to the server. Sure, the Signal servers could be slurping up IP addresses and timings despite advertising that they don’t. But I can’t think of a single alternative service that can guarantee otherwise, and that problem is compounded by clients that leak much more metadata than Signal.