But if you store everything in the password manager, it's no longer two factor.
Sure, if your password vault gets breached then everything is exposed but that's extremely unlikely and you have a lot of work to do in that event regardless. It's an inherent risk to using a password manager: everything is centralized so it's a valuable target.
The point of a second separate factor is to reduce that risk.