The point of the intermediate server is so that you can NAT hole punch, otherwise it has little point (except maybe as a relay, which is a bad solution).
Note that the transit relay only sees ciphertext (for bulk data transfers). Even the mailbox server only sees ciphertext or SPAKE key-exchange messages. No server sees plaintext ever.