Non-linear utility of money.

Going from $1bn to $50m (a loss of 95%) in wealth might be psychologically damaging but you're still independently wealthy. You'd lose the jet (if you chose to buy one), but you'd keep the house, the car, the neighbourhood, etc.

Going from $1m to $50k (a loss of 95%) is absolutely enormous. You've lost the house and the retirement fund and you're starting from scratch with a deposit.

The realistic case for most labourers is worse than that. How much risk is involved in moving to a new town for a job? Training in a certain discipline? Deciding to have children or not? All of those are far more impactful on the middle class than on the wealthy.