If every white person thinks the N-word when he sees a black person, that fact creates an inescapable environment of hostility where anyone would feel completely helpless to create any sort of positive future for themselves. Of course we don't live in a world where everyone is this blatantly racist. The point of this is to explain to you in a different way why racism is more powerful than just "you exclude people based on race." Some people need environments where they can feel comfortable learning, before they can get up to speed and have the confidence to defend themselves against criticism and hostility.
...you should really take a look at the more common definitions of racism to understand why this point of view you have in one held by a minority of people.
Right. Because letting the majority population define which terms an aggrieved minority can use to describe its own situation is completely sensible. What a surprise, the discriminated minority has a different view of the majority's behaviors than do those in the majority!