Many MFA implementations will not allow you to register more than 1 TOTP authenticator, and so that is why I typically use the trick of registering both Yubikeys simultaneously. If you wish to register another, the service will often require you to deactivate MFA and set it up from scratch, which de-registers the original one. I think this is probably mandatory, due to the way the secrets work. There's no way a service can safely store such a secret on-site, because this would defeat the purpose.

So in fact, if your alternate key/auth is offsite, then you're actually forced to manage your own secrets in order to properly register all affected devices.

You can have BitWarden or another password manager manage those secrets for you, but you've just reduced your MFA to 1FA, because obviously that's where you store your passwords. In fact, revealing a TOTP secret is more damaging than revealing a simple password. But, you do you...