This is about proving to Google that you're secure. Google doesn't know if the password you entered came from a password manager or not. But if you're using a hardware key, they know it's secure.
If you lose your hardware keys, you still have your other 2 factor options, so you are no worse off than your current situation.