A password is something you know - it's in your head, so it can only be stolen if you save it somewhere, or if there's malware on your computer sniffing it.
The tokens in the phone are saved in the phone, so if you lose the phone, you've just lost your password/set of keys. On top of that, malware in the phone can extract the keys.
With malware on your phone, your accounts can be pilfered without your knowledge, at any time. Or if your phone is stolen. This is different from 2-factor, where you have to both know a secret AND have access to a device.
(If the prospect of malware on your phone doesn't phase you, consider that news articles over three years old reported hundreds of thousands of malware installs found by various security companies)