One option is to have two identical security keys. In general, you can't easily read the secrets from an existing key, but you can overwrite/initialize them to get two with identical data.
Most services allow you to enrol two or more security keys, so you wouldn't need to overwrite one.
The problem with that is that it requires to have all these security keys available in order to enrol them, which is not possible if you want to store one of them in a different secure location. If you have two keys in your pocket, that's not much of a backup; having two identical keys means that you can enrol the one in your pocket and if it gets lost, the copy from your safe works.
Do you sign up to new services daily? I'm thinking of it more like sign up today, store the second elsewhere and don't think about it again.
Not daily, but something like weekly - I certainly signed up for more than 50 new accounts/systems in this last year, and a few of them had 2FA.
Is this possible with Yubikeys? I'd be very interested if so. "Two identical Yubikeys" is something I've wanted for a long time.