Being attacked or harassed for racial (or any reason) is always unpleasant, but I've always found it easy to rationalize such experiences as outliers, as in, "Well, some people are just ignorant assholes". So I've found the things that happen at an everyday level to be far more deeply unsettling. I've been in a few countries where Asians are rare (yes, rarer than in Iowa) and while I hadn't been physically attacked, the experience of people just staring at me, everywhere, while I'm going for a walk or waiting in line at the grocery, was so unpleasant that I just found myself just not going out at all very much. Not as a conscious decision, but from a reluctance that built up subconsciously, and something that I only realized after I returned to the States.
It's not that egregious acts of harassment don't happen to minorities today. But when I think of why so few [insert minority group here] are in [insert role/geography here], I believe it's because of the undramatic but accumulated things that the OP mentions, all of which may sound like whining to people who, while genuinely disapproving of overt attacks, don't realize that lesser slights can be enough to create a dampening effects.