Even if I do have keys, they are safe in my pocket, not sticking out the side of a fragile USB port.
There's then the whole mobile problem -- yubikeys are perhaps fine with my laptop, but how about when I'm using a mobile and my laptop is in my bag, or at home?
And OK, lets say I solve all that. How do I add a second key?
The beauty of SMS for 2FA is that my phone number sticks with me. If my phone is lost or stolen, a new sim card is sent to my home and I have access to all my 2FA authenticaitons. It also ties in well with my phone -- if I get an SMS with a number 123456, it appears as an automatic insert option on the form, no need to go to another app to copy a number and switch back to paste.
TOTP and Yubikeys do not match the usability of SMS.