this is not offline brute-force, you're not going to iterate over thousands of codes...
Passwords are permanent, that's the main difference. the code is only relevant to 30 secs after that it is useless. passwords are always useful because they never change.
Also people tend to re-use or slightly modify their password. with TOTP codes, the previous code doesn't tell you anything regarding the next code.