The real answer for "why not a smartphone app" is "because code generators are just as phishable as passwords". In the real world, that's how people are being compromised, not by elaborate phone exploit pivots but by phishing pages. It also speaks to why phone authenticators are acceptable backups to tokens.
The real answer for "why not SMS" is "because both teenagers and intelligence services can get a phone number redirected; your phone number is not your phone."
Obviously, you don't PIN-lock a U2F token; the answer to "what if it's stolen" is "whoever stole it probably doesn't have your password, which they'll need in order to use the token, so if your token is stolen remove it from your account and then fish $17.99 out of your couch cushions and buy a new one".