For most accounts your email password is already a single point of failure (password reset flow).

Even where that's not true, the tradeoff with a password manager is in strengthening defenses broadly in exchange for a more central point of failure, compared to weak defenses all around but no single point that gives everything away. The idea being that you can remember one, really good high entropy password, vs. having a bunch of weak passwords.

Of course, the theoretical ideal would be having lots of different strong passwords, but that's not how human memory works.