I was talking about the GPG agent, so that the key on the smart card can be used to for sudo elevation on the remote host. This usually requires user interaction with the key, so just having access to the agent wouldn't do much. I don't think the ssh agent would help with this.

To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.

    $ ssh-add .ssh/id_yubikey_gpg.pub                                                                                                                                                                                                                                                                                                                     
    Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto

SSH connections work fine with that key.