My irritation is that I know what it is, and I've said no thanks many times, but I'm still asked regularly by the likes of Amazon, and they usually pick a time when I'm trying to order something quick¹. It is one of the growing number of things in life that simply have no “no” option, it is always “yes or later” - I wouldn't mind so much if “later” meant “I know the option exists, I'll ask for it if I change my mind, don't bother me again otherwise”. Call me cynical, but if companies are trying to nag me into something I very much doubt the main benefit is mine. I'm sure there are many people out there who go along with it simply because they are sick of being asked repeatedly.
I also don't see the real benefit with the way things are often implemented anyway. When the credential recovery process is sending a magic email or text, making SMTP or SMS the weak link of the chain just as it often is for passwords so I'd be giving up my preferred workflows for no better security.
----
[1] A short while ago I actually ordered from somewhere else because of this, bitter twit that I am. “I wonder if I can get this almost certainly drop-shipped item on next day delivery via Prime?”, [goes to Amazon to check], [get passkey prompt], “sod it, I'll go back to the original place”.