The same is with cutting back on comfort of living. If I cut my comfort, but my neighbor don't and payoff is equal for us both, why should I suffer more than him? This is our basic species morality which we found even in apes [0].
[0] https://www.theemotionmachine.com/what-great-apes-teach-us-a...
Seriously? You mean Bill Gates works because otherwise he'd starve to death?
More seriously, I challenge you to hold a poll among your friends - given $x per month where x is enough to survive, would they stop working?
Everyone I speak to about this says that he/she personally would not, but the "other people" certainly will.
Not working means that you reclaim 50%+ of your awake time, and not any 50%, the 50% during which you spend most of your energy. I'm already living on ~70% of the min wage of my country, and I don't feel like I'm restraining myself at all so it wouldn't be such a big deal.
One might argue that this is all work.
2. You and your friends are probably not like the average person.
3. You were probably raised with the idea that it's very important, both financially and socially, to have a job. It might be a lot harder to raise your kids with these values if they grow up in a world where working or not working has no financial impact.
A farmer might still plant, but decide to plant flowers instead of food. Or maybe they would plant exotic fruits instead of staple foods.
A programmer might still program, but they might decide to play with algorithm problems instead of developing medical software.