Want to go hiking? Swimming in a river? Build some furniture? Build a house? Watch birds? Go shopping? Wrench on your car? Race your motorcycle? All are best done during the "work hours".
So while I can see the appeal of not tying one's meaning to one's job, I also completely understand the sinking feeling of having to postpone these activities for doing... nothing.
Because even though you can "find something to do" while at work, most of the time, you're not fully free. You may not just pick up your stuff and go fishing. You have to stay physically at the office and sometimes even have people dropping by asking random questions. So even getting engrossed in a book / music is out, since you'll be randomly interrupted. At least if you're WFH you can do random chores to free up the time afterwards (do the dishes / laundry / bake a pizza).
Of course, if you have to work to live, you have to go through this. But I don't see how its being necessary can make it less soul-crushing. Quite the contrary. I'm lucky enough to be able to find things to do on my own, or if not, work for one of my hobbies which happen to involve computers and don't require me to be physically somewhere precise. But not everyone's so lucky.