If you use Google’s backup for WhatsApp, it is NOT encrypted. Your local backup IS encrypted. It’s not your data - you can’t have it except through the app. But google can (and does) have that data for likely 95% of android WhatsApp users; E2E with this state of affairs is not very useful. (I assume the iOS situation is similar, but don’t know for sure)
Also, a copy of your phone book, continuous IP (and likely location) and all your conversation metadata is stored on Facebook’s servers even if you don’t use the google drive backup.
Yes, WhatsApp is useful and works well, But the E2E aspect is false safety. For many practical purposes, you should assume there’s No encryption. For the purposes that inconvenience you, it’s obviously there.