Every time this pops up, I feel compelled to point out that in the 8 years I've lived in Japan (over a 10 year period), I haven't found this to be true.
There is racism in Japan, just like everywhere. When I lived in Canada, I saw a sales manager where I was working throwing how CVs with Indian names. I asked him what he was doing. "There's no point in hiring someone whose name I can't even pronounce", was the reply. In the UK I couldn't get on a bus without somebody complaining about how dark people were taking over the country. Racism is everywhere.
It's a good point that in Japan you will almost never run into racism in polite company -- because, as you say, it is considered impolite. However many expats who live here complain about racism. What's going on?
IMHO it's not racism, it's culturalism. In my experience, if you speak Japanese well and you know how to act as a Japanese person, you will see almost no racism. However, if you are a visible minority (as I am) people often expect that you are ignorant of Japanese culture. They put on the "you are a guest" routine, which is fine except that it wears thin when it is your home. One or two gestures and a word or two of fluent Japanese almost always snaps them out of it.
Just to illustrate the difference, I once went to Takayama with my wife (who is Japanese). We stayed at a traditional ryokan (inn). Takayama has a very famous festival and fairly large old town, so it is popular with tourists. Because my wife changed her surname to mine, the woman running the ryokan assumed she was not from Japan (although she is very obviously of Japanese ethnicity). It took quite a while for the woman to twig that my wife was, in fact, Japanese and lived in Japan. It was the first time my wife experienced that difference and it surprised her quite a bit.
I have experienced some racism in Japan, but I really don't think that it's all that much different than anywhere else. Sometimes I think that people don't see the racism where they live -- often because they are not a visible minority. When they become a visible minority for the first time, it's a shock. When I went to University the CS department was composed mostly of foreign students from China or India and most of my friends were from there. It was the first time that I really noticed the racism in Canada.
I've run out of time, but at least in my experience, it is relatively easy to live in Japan if you decide to be Japanese. They don't do melting pot here. If you want to hold on to your own culture and to act like you did in your home country, you're going to have troubles being accepted. But if you decide to accept Japanese culture completely, I don't think you will run into any problems. The if part is hard, though, and I've seen many people run into the brick wall that is Japanese culture.