Better than having then unencrypted (unless you set a passphrase) on the filesystem for every rogue process to grab. The issue with password managers is the key is then most likely still accessible from other processes while the password manager is unlocked. On linux for example by poking around in /dev/mem or /dev/kmem. There are way that help with that like memfd_secret. A secure enclave or similar side processor avoids this issue by running within its own isolated memory and this processor also handles all key operations, so the key never enters the main memory.
Yubikey would probably be a better alternative.
Any idea how to set my YubiKey up to handle this?
If all the servers you talk to are up to date enough to accept SK keys, I would use one:
I can recommend this guide [1] to setup your yubikey with gpg, which allows you to both use it to sign or encrypt with gpg and also to use it as an ssh key.
The readme specifically mentions Yubikey support. Given the restrictions with Secure Enclave (no import, export, or backup), a smart card or Yubikey probably would be better.