As long as it has nothing to do with work performance or other character traits, I don't see racism here. Otherwise you'd need to ban any discussion about appearances at the work place.
It's perfectly reasonable to adjust coming anywhere close to discussing race as a strategy to avoid getting fired, but it's insane to redefine racism based on that.
We're doing that, and it results in a constantly expanding definition of racism, to where there are now "microaggressions."
Keep in mind, if you're white, you don't care: as you suggested, you just avoid talking about certain topics and carry on.
If you're a minority group, all these white people are, against their own apparent interests, agreeing that there's pervasive racism everywhere. The minority person has to live believing that all this is true.
We ought to have empathy and get this right so we don't contribute to a pervasive falsehood that makes people's lives seem worse than they really are. What's no big deal for us can be a ton of stress for others.