The BIG difference is that you have to trust the hardware and the operating system already, and as these are made by apple, you already have to trust them.

"Trusting the servers of a non E2E service" is adding another trusted party.

If you don't trust apple, you don't have an iPhone.