People like Marshall Sahlins (Graeber's mentor) woudl strongly disagree that leisure like this is recent, in the sense that he believe(d) that pre-agricultural human societies had lots of free time available to (almost) everyone.
The problem is the same as with gorillas: habitats supporting such leisurely life are rare, and cannot support density required for any kind of technological and most forms of social advancement. This makes life leisurely but full of peril.