I had basically no truly "negative" experiences based on my race. I am fluent (understand 95% - 98% of what's going on, including complex convos about tech stacks and so on (I work in software engineering).
Sometimes people speak (usually not very good) English to me, assuming I don't understand, but as soon as I reply in Japanese they talk to me like they would a normal Japanese person. One could consider this a "negative" experience (why judge whether I can speak Japanese based on my skin colour??) but I don't think they have negative intentions, so I don't take it personally.
I think living here without speak good Japanese would be a pretty mediocre experience. There are tons of awesome tech meets all the time, but noone speaks English. Living here but not being able to live with the people here would be a pretty lacking experience, you would really miss what makes Japan Japan (the people and culture).
I would be happy to raise a family here (I don't think I will, but I wouldn't be against it either). At least until high school age. The Japanese university system does seem nearly as good as my country (Australia), though, so I would my kids to study overseas. I think Japan is a safe place and good environment to raise a family, and has an excellent healthcare system.
One thing I noticed is developer salaries (at least if you work in a regular Japanese company, like me) are very low - I earn 50% of what my friends back home do). Maybe if you work at a company with many ex-pats you can have a higher salary.