If you've found yourself at the end of a bad economic bargain (which is most people), you tend to rationalize your ambitions and switch attention away from work to more fulfilling things outside of work
I know few people with a "rich, full life", and a lot more people who work all day, buy fast food for dinner, and then play League for 5 hours until they fall asleep. It's not fulfillment they're after, it's distraction, comfort, escape. Self-medication comes in the form of drink, food, tv, gambling, drugs, even other people.
Using your example, some seek family, a close social circle, a white-picket-fence life. There's nothing wrong with that, but another group seeks relative solitude, doesn't see a purpose in creating a family or having a nice house, and fills some kind of intellectual hole, be it with creative writing, or relatively non-conformist research.
My point is, I don't see Bukowski as one who lacks empathy, rather, someone who follows his own purpose, though, potentially blind to some others' happiness doing other things.
Lastly, the workforce ethics back then were quite different from today's. It's my understanding that the pay was significantly lower compared to costs of living while progressing in ranks was near impossible without a degree.
He's describing the people he saw every day for decades. If you think you understand those people you've never met more than he did, you're the one who lacks an understanding of empathy.
If society wanted to be more humane and empathetic, it would force all young adults to be homeless and impoverished for some time, and working three jobs to get a taste for how it is for everyone else that isn't sheltered or privileged. The golden rule and empathy isn't practiced as much anymore because there are greater convenient distractions, selfishness, ignorance and lack of community that has the effect of atomizing people from each other... we are mostly divided-and-conquered and enslaved.
His further writings talk about escapes hes tried: drinking, gambling, abusing women. He was abused himself. He didn't have an easy life. He was homeless. He won the proverbial lottery but that didnt let him escape himself. Not fully.