Most folks can't or won't do lots of things in their lives (e.g. plumbing, electrical, construction, lawn services, Automotive).
The main thing blocking routers and IoT devices is the control every vendor wants to hold over their customers' devices after sale.
I'd argue that the main blocker to IoT security is the lack of culpability on the part of device manufacturers. I don't want to go so far as to suggest that companies should be wholly liable for software bugs, but vulnerabilities that are brought to the attention of the company privately or disclosed publicly absolutely should be their responsibility to address.
For you or me (or most of the folks here I suspect) we feel better if we had the ability to decide what software our fridge runs, but for 99% of people they're better off if their fridge's manufacturer provides them with regular security updates for the life of their product.
That being said, these aren't mutually exclusive. In a perfect world we'd have laws compelling fridge companies to allow 3rd party software if they don't keep their firmware up to date.
Even if I don't buy one, I dread to think of what might happen if enough fridges across the world stopped working all at once (demand for non-perishable food and fridges would skyrocket).
For the sake of consumer safety in our imperfect world, there should be a safe-mode hard switch fallback for any life-critical and/or high wattage device that gets networked.