Yes, but passwords can be keylogged and SSH keys on a hardware token (like yubikey) can't. Also if you've got touch-to-use enabled on the token you can't even use the key without physically touching the token.
AddKeysToAgent defaults to no. Ptrace might also be disabled, depending on system. I would be more concerned about keyloggers, or any tricks that result in me running something else than the real ssh client (e.g. custom program somewhere in PATH).