There's probably a less confrontational way to phrase the question if you're looking to have a meaningful discussion.

Are you confused by the concept of affirmative action? It's not a new idea and there's a considerable body of academic research and opinions for and against it online.

Group X is not well represented, due to many factors. We can't address those factors, but we can help Group X in our organization.

Racism is a pretty heavily charged accusation to make, and you made it very casually. I believe you weren't trying to troll, but rather than repeatedly saying "I don't mean to be a troll", you might have better luck by being more careful about how you phrase your concerns. That's all.