For example, https://github.com/str4d/age-plugin-yubikey makes it very easy to use PIV tokens, including YubiKeys, with age. (Well, for now with rage, since plugin support is coming in age v1.1.0.)
I argue against password-protecting keys by default because, unlike using hardware tokens, it doesn't protect against many threat models.