Replace the word southerner with some other minority and see how it sounds. Not all southerners are racists, and inbred is just a slur.
That said, the political climate is a real concern for me, but I'm hoping it wouldn't be in my face in a tourist town.