Nobody likes crowded or dirty space, but an aversion per se to having other people around is definitely pathological.

I think this is a social trust thing - lots of people are just scared of strangers. This is particularly true for high-violence/high-tension societies like the US.

My city (in Australia) is very peaceful and I don't feel bothered or frightened at all by the general public - in fact I like being able to people-watch, feel a certain sense of vibrancy, etc.