Direct communication isn't required to create a hostile environment. Sly looks, offhand remarks overheard while trying to do an exercise, etc.--they pile up. And poor people especially are conditioned to be mindful of this sort of thing. It's often dangerous (as in, an altercation that can become physical might occur) not to be when in public.