You're not thinking on a long enough timescale. Hunter gatherers only worked 10 hours a week :)
How much time was spent in meetings, how much time preparing food, fixing clothes, patrolling territory, scouting things out, making home improvements etc.
If the modern person only worked for food, then I think a few hours a week would suffice.