1. without a password manager that is shared on all your devices, you WILL re-use passwords out of frustration. 2. without a password manager, if you do any sort of regular sharing passwords with a engineering team, friends & family, you'll resort to pretty insecure channels. 3. true E2E encryption, while still providing some surface area, has proven in the field through multiple pretty bad breaches[1], that it's a security model that holds up under real-world circumstances.
On the flip side, you are right: you are one compromised browser extension / binary away from having your local vault decrypted, and ALL your passwords compromised. But think about this: if someone has this much local access, chances are they can install a keylogger anyway, or read your clipboard, so the real difference is you've conveniently pre-loaded all your sensitive information in one go for the bad actor.
[1]For example: https://blog.lastpass.com/2022/12/notice-of-recent-security-...