Phone numbers though are terrible because they're tied to countries, their security depends only on your carrier, you can't run your own carrier to take it into your own hands, and sending SMS costs money. Also the underlying interconnection networks like SS7 aren't secure at all and rely on trust.
TLDs are managed by governments or government-adjacent organizations. Domains are managed by the TLD manager. Email addresses are managed by the domain manager.
I've never had a phone number or a domain name taken from me, but I've heard of more cases of the latter than of the former.
I've heard stories of how a US carrier issued a replacement sim card to a fraudster. I've also heard stories of how a Russian carrier intercepted someone's SMS messages to break into their Telegram account.
SIM swapping is a big problem, but that is more equivalent to DNS hijacking of your domain than to the concept of legal/contractual "ownership" or rights to operation.
But semantics aside, the important question is how easy would it be for a malfeasant to interfere with your quiet enjoyment of your public identifier (phone number or email address). SIM swapping is way too easy. Domain hijacking isn't hard either, and sometimes one attack can leverage the other.