The error in this is the thinking that the gains from agriculture accrue evenly to everyone. In reality, the "us" who are farmers and the "us" with leisure time are not the same people. You can see this by looking at the economic stratification for every single agricultural society... ever?
I would highly recommend reading James C Scott's 'Against the Grain' [0].
His thesis is that agriculture came in quite slowly, as part of a semi-sedentary hunter-gatherer society. It was only once the natural environment was depleted and agriculture became the only option to feed yourself that fixed, sedentary agriculture took over.
This is all tied to the rise of states, which could only obtain stability once their taxation base was locked in place. It's the best explanation I've seen for why "we" would choose agriculture.
[0] https://www.goodreads.com/book/show/34324534-against-the-gra...
(Incidentally I think it's much better written than his first book 'Seeing like a State', if you've read that.)
His viewpoint (while interesting) was not very grounded at all. Simply because agriculture requires more work doesn't mean it's inferior to hunting. Try hunting an antelope if you've got a sprained ankle.
Another perspective that irked me was his blind faith that modern society is built purely conceptual ideas. Citing empires, car brands, and money. In reality all these things work because they provide practical benefits. Few people or groups go out of their way to work purely on ideological bases, and those who do, do so on a rational philosophy. Let alone as soon as it becomes impracticable (See Maslow's hierarchy of needs).
* Worse nutrition because you're eating only the narrow selection you can grow.
* Poor nutrition means worse health, digestive, and dental problems.
* Agriculture leads to higher population densities and living in proximity to animals which in turn leads to plagues.
* Agriculture leads to population growth and consequent vulnerability to weather, floods, etc.
* Agriculture leads to population density, leads to social organization, leads to war, genocide, feudalism, etc
There are lots of problems with primitive agriculture and it seems perfectly plausible that it would be a better life to be an early hunter gatherer than an early farmer. Try tilling a field with a sprained ankle, or with rotting teeth after a bad harvest to feed your starving children. Etc.