If you hand out your currency to a third party (Joe Binance or someone else), then it can "easily be stolen", regardless of what currency your are talking about: using dollars instead of BTC changes nothing here.
No such back-stop exists for a third party stealing your bitcoins.
In this case, we are not talking about a bank, but about an exchange which exists outside of US jurisdiction. In this case, it does not matter if you handed USD or BTC to this third-party (outside US jurisdiction): if they decide to take your stuff, there is little actual recourse you have and FDIC won't cover it.
On the other hand, if you are dealing with an exchange within US jurisdiction (e.g. Coinbase), I don't see how BTC theft would be treated any differently from USD theft: if they take your assets, they can be brought to a court to have that fixed and return your assets (be it USD or BTC or whatever).
TL;DR: What matters is if you keep your assets (USD or BTC) with an appropriately-regulated institution (e.g. a bank within a jurisdiction you trust) or not (e.g. in an unregulated exchange outside your jurisdiction), and not so much the type of assets you have (or that were stolen/taken from you).
(and perhaps even further than that, my earlier point is, "Joe Binance" is in so deep at this point that there's now likely enough of an incentive for parties who aren't "legal" to threaten him with violence or harm if he does something wrong.)
Recovery can take weeks, months, years, or be impossible (if the cash was burned, or the BTC routed to an account with a lost private key). While all that is going on, FDIC insurance on USD means (in the case of a rifle US bank) the Fed steps in and issues new cash to make the victim whole.
There is neither a mechanism for printing "emergency Bitcoin" nor a "victim's reserve" operated by a trusted third-party to do something similar if a Binance (for hypothetical example) goes rogue.
If you are talking about decentralized tokens/coins (BTC), then... yes, you can't arbitrarily seize or print new ones (but that resistance to arbitrary manipulation is generally considered a feature, not a bug).
And, again, in this case (sending money/assets to Binance, or even to a US-based exchange like Coinbase), FDIC would be irrelevant and not be triggered, even if you use USD (I assume it only applies to money kept in banks and not necessarily to money kept in other type of non-bank brokers or exchanges).
... but worth noting: I'm not talking about recovery (which is a matter of law that could take weeks or years). I mean replacement: making the victim whole with new cash while the law addresses the theft itself. I am unaware of any process or institution in the Bitcoin space that can do that.
That brings up a really interesting notion, no? Couldn't be done with bitcoin, but one might imagine one of these more centralized alt-currencies doing something like that. Somebody gets openly and obviously screwed on Binance and Joe Binance reviews/audits, and says, yep, and issues them some Binance coins*
*I know Binance does have it's own coin (coins?), I have no idea if it's technically possible with it/them, this is purely hypothetical.