Yeah, sort of a myth, even pre-constitutional.
Rhode Island was founded because the Puritans quite like what Roger Williams had to say. Ben Franklin was noted of making a remark about there being too many Germans. The Potato Famine brought Irish emigrants and with it slurs ('mick') and hatred against that demographic. The Southern Italians weren't considered white when they came around (also had their slurs coined, 'dago, 'wop', 'guinea').
A sudden (or continuous, but seemingly overwhelming) influx of an overwhelming amount of people who are "new" upsets the ecosystem. If me and, say 30 of my (relatively) leftist, mostly-holding-graduate-degrees friends (and their spouses, etc) all moved to a small-town in West Texas, surely we'd be looked upon as differently. It wouldn't be because I had brown skin (which I do, and there might be a little bit of 'hmm that fella looks kinda strange' at first). I can tell you from contracting all around the world from rural Alabama to Beijing, that the reason my buddies and I would be singled out would be our mannerisms, our speech patterns, our dress, our modes of communication, and what we consider polite. (I.e., when I'm in NYC, I may see a friend sitting having a coffee al fresco. He's busy on his iPad, I'm late for a meeting - I'm not going to stop and say hello. Contrast that to someone who might take that as a snub and potentially harvest resentment for years.)
This allows me to segue onto a tangent and make an important sociological distinction (it's somewhat relevant, I swear).
Lets say grew up in a small town in Alabama where your pa and all his kin have been for damn near 3 generations. It's completely reasonable to be slightly off-kilter when an aggregate of people disrupt the stability of your living situation. Your memories were formed in that town, you're invested in all sorts of ways in that town - emotionally, financially, you name it. Your memories were formed in that town. The money you made from mowing lawns when you were 12 (do kids even do that now, or do they just Instagram and yell at their parents?) was spent at that local ice cream shop. You had your first kiss in the back your dad's car after you and that girl shotgunned a joint in the school parking lot cutting class. There's still a blood stain on concrete of your parents houses driveway from when you hit on that jock's girl and he came around and broke your jaw (OK, that one didn't happen to me, but you see what I'm getting at, I hope).
That doesn't make you inherently racist, it makes you upset at the prospect of losing some stability you used grew up with and expected (however reasonable or unreasonable those expectations were) to have. Now, if I come around with my buddies, we buy your parents house, knock it down and throw up some ugly McMansion, get a 30 year lease on that ice cream shop's property, knock that down and replace it with a raw-food vegan or fair-trade we-only-serve-black-coffee-here cafe. You'd be upset at the culture shift and not because some brown dude did it. There are certainly racists out there, there's been this media depiction of any Trump supporter being one. This is problematic because it not only does it mask the real issue (there are active neo-nazis out there, burning crosses, and sitting around hating everyone as their Friday recreational event) but it divides the country even further down political party lines.