Why? Your GPG key is airgapped and inextricable (except possibly via something like a DPA-style attack), and you can set a PIN required to perform any private key operations, complete with a configurable number of attempts before the device wipes itself.
There are DPA attacks to be worried about, but if you set a decent PIN and limit the number of failed attempts your GPG keys will probably be safe if your Yubikey is ever lost/stolen.