Here's an example of what I consider a reasonable hardware key security model:

I can initialize a Nitrokey HSM on a disconnected PC with whatever device key encryption key (DKEK) I want (128-bit AES key). I can write that key down and store it in a safe place (broken into chunks, if need be, for my security model). I can backup my keys generated by the Nitrokey to export files that are completely useless w/o my DKEK. I can store those backups wherever I want without any loss of security.

If I lose my Nitrokey device I can buy another, initialize it with the same DKEK, and restore my key backups. (If I really want to I can decrypt my backups using my DKEK.)

If I don’t want to do any of that I can have the Nitrokey generate a random DKEK that is never exposed. Then I lose all my keys permanently when I lose the key. I have the option to choose the model that allows me to backup my keys if I want to.

If the point of the hardware key is to put the owner thru pointless make-work and risk then the hardware key is serving somebody other than the owner.

No. The backups have their own secrets and no relationship to one another. You register them as distinct keys with each app.