Aside from situations where high security is needed (which would necessitate a dedicated piece of hardware to be the 2nd factor), Why are you adverse to using a smartphone as the 2nd factor?
The problem also is that the smartphone can be hacked and opens a huge attack vector that you don't have with other systems.
Also, typing in stuff by hand is a general bother.
There is also the matter of control. At the moment there are only a handful of smartphones that run an OS that is not under the control of Apple, Microsoft, or Google (Ubuntu smartphones come to mind as an alternative). On the other hand, on a normal computer I can run a full free software stack. To me this seems odd. If I own a piece of hardware, I want to run an OS and applications I can legally modify and tinker with, especially if I use it as a security device. By using tools such as Google Authenticator I get sucked into this walled garden.
The behaviour of hardware tokens such as Yubico's U2F keys on the other hand is completely verifiable — mostly because they are quite simple compared to a smartphone.
(Aside from these personal views on software, I do have high security requirements as well for some services, where a dedicated hardware token is preferable, as you point out.)
From a software freedom perspective, hardware tokens/smart cards are no better than smartphones.
I agree that the ideal situation would be a device that is completely under the control of the user, but compared to IOS or Android, a hardware token that only does what it is supposed to do (in the case of the U2F tokens; register and sign authentication requests), and cannot be modified by the manufacturer after sale is very much preferable.