Would you rather them not invite you? I've always found that rich people are not trying to put you down by talking about their boats, clothes, or businesses. They just live with a different set of parameters that people can't relate to.
I think it's wrong to expect Harvard to provide a "safe space" for poor people. We should instead expect everyone to respect each others differences. Rich, poor, jamaican, colombian... People are people. Instead of being self-conscious about what makes you different, embrace the differences and expand your world view.