And besides, the towns that do get/have these institutions become gentrified and all of the the original residents get pushed to nearby small towns by professionals from more affluent regions.
In the end you end up with an upscale, expensive small town and surrounding depressed blue collar towns.
Source: I’m from a small town with a university.
Perhaps the assertion of racism isn't quite true, because to be really complete, you'd have to add ageism.
It’s also not the same as saying “I don’t want any old white people in this town”.
I mean, it's a racist personal preference. Caring about whether the expression of it is racist seems besides the point?
Among other things, it is certainly socially acceptable to date or socialize mostly with people inside your own race, and I don’t know of many people who would hold this behavior to be racist.
EDITS: wording, clarity. Apologies.
Gladly throwing myself in the pedantry to avoid the real discussion, the only reason we listen to other people at all is that their words are related to their thoughts and their actions. So, the reason you'd admonish someone for saying something horrible is that they indicated they were thinking something horrible, which gains status as horrible because it congruent with a mind that would do something horrible.
That said, I still do not think I would call it racism. Most of us prefer to be around people our own age when given the opportunity—for shared culture, lifestyles, etc. Is that ageist? I really don’t think that bigotry really applies here. It’s much more useful to apply solely to actions, which are easier to reason about. Discriminatory personal preference can certainly be harmful, but it seems like a distinct social phenomenon to bigotry.
More pedantry...
> So, the reason you'd admonish someone for saying something horrible is that they indicated they were thinking something horrible
No, the reason would be fear that they would act on their thoughts, or because their speech did affect people, not that their thoughts are themselves immoral.
Edit: To complete the pedantry, note that the end of my sentence brings it in line with the content your objection to the start of the sentence: >which gains status as horrible because it congruent with a mind that would do something horrible.
Similarly, it's not heterophobic to say "I don't want to work someplace where they don't hire gay people".
Agism or not, it's generally true that small towns with a lot of old people are typically towns in their death throes and white enclaves are typically practicing some form of racial exclusion.
Restaurants that serve cheap-eats (e.g. Chinese restaurants) tend to be ok since they have low overheads and are designed to handle peaking volumes easily (stir fries are easy, and most are family owned). But shoe, book retailers etc. will suffer in the summer when school's out, and in the winter when nobody's out and about.