The problem is the well intentioned but naive opinions that are upvoted and popular. Do you remember the popular comments from the GitHub diversity posts? Any diversity post for that matter.
The demographic here is mostly male. In a homogenous environment, regardless of which combination of race, gender or anything else, people have blindspots. This remains true when the majority of participants are decent. For example, a group of polite dudes can create an environment full of casual sexism.
Lastly, I flag insults and one line jokes. Maybe you can help us understand what gets flags and what gets downvotes.
However, I am happy the comment is removed. You are right, maybe I don't give the moderation enough credit.