I experienced something similar and would posit that small communities accommodate diversity more easily, because you get to know the people, it's no longer "the homosexuals" or "the immigrants" or "the jews", it's "John who works at the coffee shop ".

Clearly this breaks down at a certain size, and it may still suck for people on the minority side tho.