You can't completely remove the trust factor no matter what you do. Can you completely 100% trust your computing device? Maybe. Can you 100% trust your network? Maybe.

But things like encryption help build trust. They ensure that there are safeguards in place in case something gets compromised. They reduce number of bad actors from unlimited to just one (the vendor).

In my opinion, all these are very good reasons to always opt for E2EE even if it isn't perfect.