I'm confused, what is it that you're trying to say? None of it adds up to "you are trusting the server" and all of it (except the part saying that you're necessarily trusting the server) is exactly my understanding.
Some of the other designs you described try to have users verify the ephemeral end-to-end encryption keys. If Signal did that then obviously each call, text conversation, or whatever has new keys, trust doesn't continue from one to another. But Signal's long-term identity key relates everything together. The Safety Number is about ensuring you really have Bob's long-term identity key (and Bob has yours) rather than about this particular call, conversation, etc.