Because people have an innate sense of right and wrong. The concept that you should not kill someone else in order to take with they have has been shown to be almost universal across time and culture.
I doubt any poor person sees a rich person as a part of their extended social group. The psychological barrier does not exist.