This is why I use two factor authentication for my email. It's a usability nightmare, but not as much of a nightmare as losing all my accounts everywhere.
Security is only as strong as the weakest link. CloudFlare was hacked recently because the attacker was able to redirect voicemail to another account, then use the two-factor backup recovery phone option to take control of Google Authenticator.
https://blog.cloudflare.com/the-four-critical-security-flaws...
Hence, a usability nightmare.