Yeah, we use Yubico yubikey nanos at work for ssh 2fa as well as gmail 2fa. It's pretty nice.
Or there is another attack vector that I'm not aware of?
I imagine if somebody steal my password I would get notified (Gmail) and could easily switch to a new one, no damage done.