I don't buy this construction of the workday where spending 50% of your awake hours at work leaves people so exhausted they can't do anything else with their lives, but if we changed that to 38% of their waking hours they'd be so bored that they be starting businesses and volunteering all over. That's not even consistent with your own experience of being exhausted halfway through the work day. Two extra hours per day isn't going to translate to launching a new business or volunteer effort.
You hinted at the real problem: Most people don't have the time management skills and motivation that they think they do. Remove a couple hours of work per week from most people's lives and those hours will get redistributed to mostly leisure time. Some of it more productive than other options (socializing with the community, working on hobbies).