A lot of the people I meet who are less concerned with these problems do so across the board - unless it directly affects them or they've personally lived through those exact same issues they cannot (or will not?) empathize. It takes direct personal experience to change their minds. I think people who won't/cant empathize have a more simplistic view of the world as well - if you are poor you are lazy, everyone starts relatively similarly and what happens to you is mostly (if not all) related to the effort you've put in because maybe that has been their personal experience. I've been lucky enough to relate to this because I haven't had anything directly holding me back (that I am aware of), so my success or lack of success is more directly attributable to my choices and work ethic than people in worse situations than me but that doesn't prevent me from seeing how circumstances dictate choices and opportunities.
I also think that there is as psychological aspect to this in that a lack of empathy also allows people who are not poor or homeless to feel as though they achieved it. Even if they are lower-middle class average people, they've achieved more than someone who is truly poor or homeless but this is just from personal observation, I think you have the same types of "self-made-person" feelings at every level of the socioeconomic ladder.