I don't think end-to-end encryption prevents server backup. It means that decryption keys is in your mind and at most on your personal device.
This helps against everything except a valid search warrant from your government. If you don't do outright illegal stuff, and don't live under an authoritarian regime, it should be fine. (If you do both, you have bigger problems.)
Without some easy to use out-of-band key backup solution, an E2EE server backup is no backup at all.
> After losing my phone and not having a way to recover a lot of data, I've come to the realization that I don't want end to end encryption.
Effective backups and key management are 100% related to this thread
If you have an Android device you can literally backup your Signal message history and move it wherever you want.
I think Windows has an unofficial option. But yea you're out of luck on macOS and iOS it seems.