I think it's more complicated than that. I'm lucky enough at this point in my life to have enough money to afford a nice house and nice toys, and not worry much about money. I also have absolutely zero issue going to hang out with friends who have much less than me. I really don't care at all. Some of my happiest years were when I was younger and poor. I had just barely enough money to make ends meet most of the time, was living in a back room of a dirty little house with three other roommates, and I was living happy because I liked who I was with. I get that money and real worth are two very different things and enjoy people for who they are, not where they're at.
But sometimes I can clearly feel the awkwardness coming at me when I hang out with people who are significantly less well off than I am and they know it. I'm usually not sure how to handle it, and typically end up just not hanging out with them much. I sometimes also find similar situations hanging out with people who have a lot more than I do. With some people it works fine, with others I can tell by what they say and how they act that they're uncomfortable because of the difference.