The Yubikey NEO (and the 4, but the 4 is more closed-source than the NEO) is basically a Javacard smartcard with OpenPGP and PIV (x.509) applets with standard interfaces. I have my GPG auth/sign/encrypt subkeys on it (master/certification key stays offline), and with gpg-agent it will present your GPG auth key as SSH credentials.

There are Yubikey-specific PAM modules you can use as a second factor for logging in locally, and there are probably ways to use standard smartcard authentication for login purposes as well but I don't have experience with that. I mainly use it for remote SSH login purposes.

You can. The Yubikey can store a hash value in its second slot to act as a hardware token. I used to use mine with PAM so I couldn't get into my user account without it: https://developers.yubico.com/yubico-pam/

Yes, with a bit of setup and depending on your OS.

My problem with hardware tokens is simply that I lose them.