1. A somehow convinces the remote party that the actual public key is X' instead of X. Ciphertext comes in, attacker decrypts it. Done.
2. A intercepts ciphertext, feeds it to Yubikey for decryption, and gets the resulting plaintext over USB (or whatever). Note: this is assuming that the Yubikey doesn't have encrypted filesystem support or similar.
Combining 1&2, A can use the Yubikey as an oracle to perform unlimited authentications, signing, and decryptions.
The only advantage provided by a Yubikey is that your keys cannot be remotely exfiltrated. Physical attacks on the hardware are still possible though.