There is no need for this “server somewhere”, and APNS can’t have your message, mainly for two reasons:
First, that “server” is Signal server itself. When a message arrives at it, it simply commands Apple to wake up your app, for it to call home, and that’s it. It is preconfigured to do that (and GCM as well).
Second, all Signal messages are end to end encrypted and neither of Apple, Signal, Google, your ISP, any server on the route of it NEVER know its content, because the only mean of decrypting it resides in a memory of your client app. There is basically no way to see the contents before your client receives the message, by design. Even if that were not the case, various IM servers don’t have to send your texts to Apple, only if they want the text to be shown on a notification itself.
The only theoretical concern here may be that Apple can somehow pick the decryption key out of the app’s memory (because ios is a supervisor, obviously). But that can’t be the main reason someone swithces to android, which is a supervisor itself, and where the same issue exists.