Thanks for the response. I think I wouldn't be too sensitive regarding being treated different but I would feel more sensitive to how my wife and kids experience it. It's the same issue keeping me away from places in the US like Idaho and Montana.
I don't know what your profession or situation is, but if you can line up a job you can pretty easily get a work visa (it's much less binding than US work visas). Japanese corporations are known to have pretty brutal work culture so that's something you ought to be aware of when searching, if you seriously pursue it. Hope it's interesting food for thought :)
I have no doubt there are racists here, especially in the rural areas. But I also have no doubt that the same holds true for places like California and Washington. You may be keeping yourself from experiencing areas of the country that you may love and would love you back.