That is empirically false, and one's skin color is never in isolation so its irrelevant. Everyone is subject to unconscious implicit bias (not just racial). http://www.bbc.com/news/magazine-40124781
I'm specifically talking about how another person's skin color would affect me, not how I might change my behavior of them based on their skin color. Their skin color does not affect me. Their skin color might affect how I treat them, and my skin color might affect how they treat me, but their skin color does not affect me.
"I'm specifically talking about how another person's view would affect me, not how I might change my behavior of them based on their view. Their view does not affect me. Their view might affect how I treat them, and my view might affect how they treat me, but their view does not affect me."
It's no different.
Expressing their view took action, and their actions can affect me.