Self hosted Immich doesn't need end-to-end encryption, and the lack of it enables a number of very useful server-side features. If your end-to-end encryption has not undergone a security audit, it's as good as if there was no encryption at all.
Yes, that was kind of my point. Self-hosted negates the need, but most can't self host.... so that leaves end-to-end encryption the best intermediate step.