We designed the plugin protocol (
https://hackmd.io/@str4d/age-plugin-spec) and generally the age recipient/identity structure specifically to enable the use of hardware or remote keys!
For example, https://github.com/str4d/age-plugin-yubikey makes it very easy to use PIV tokens, including YubiKeys, with age. (Well, for now with rage, since plugin support is coming in age v1.1.0.)
I argue against password-protecting keys by default because, unlike using hardware tokens, it doesn't protect against many threat models.