Anyone who focuses purely on hours worked or misery while working is merely perpetuating this country's unhealthy Protestant style relationship with work. If an employee can be fully productive to the demands of their work and still have time to chat or go to the gym, what is the issue? You act like you've just described the worst thing you've ever seen ("spoiled apples") but I just see a happy workforce.
Of course, if they are truly not productive, that's something different. But unless they are your reports, you have no idea what their true level of productivity is.
It's beyond time we start recognizing that happy employees are more productive, and stop shaming people for daring to seem happy at work. Because right now, a lot of us seem like the proverbial crab not letting others crawl out of the bucket.