Ask HN: What drawbacks of using 2fa codes as main auth method?

Online services with security in mind are using password+2fa to authenticate users. What are drawbacks if I'll run online service without conventional passwords and use only time-based one-time passwords for authentication? Suppose user will be able to restore access via second channel (sms, reserve codes, etc) if passgen app is lost, so main question is: how this approach is secure/reliable?