Signal would grow immensely in my eyes if they made the server code open source [SEE EDIT], and allowed an easy way to set the centralised server address in the Signal app. The current server would be the default, and perhaps changing the server would be hidden in Advanced Options for now. But the capability would be there. That way, if x government chooses to go after Signal's servers, the availability of all the code needed to run a replacement is already available.
As it stands, we are placing a lot of trust in the Signal Foundation for communication (assuming we rely on the Signal app/service). I have no problem with a central server being used for communication if I can communicate data in a zero-trust way. But I want assurance that that server can be quickly and easily replaced in the event it goes down.
EDIT: I stand corrected. Server code is here: https://github.com/signalapp/Signal-Server Unfortunately, you cannot change the server address in the app as downloaded from any app store, so my criticism remains.