Comment A: Muslims are evil.
Comment B: Christians are evil.
It's a terrible idea to treat Comment A and B differently. The same applies whether they are talking about religion, gender, race, nationality, or anything else. You have thoroughly failed when you have built in discrimination into your content moderation system.Set A: A representative sample of jokes and stereotypes about black people found on the internet
Set B: A representative sample of jokes and stereotypes about Scandinavians found on the internet
Why on earth would its prior for "stereotypical Scandinavian" being potentially hateful be the same as for "stereotypical Black person"?
(And that's before you get into a model likely being deep enough to also draw inferences from the prevalence and content of material about the existence and impact of hatred of black people and Scandinavians respectively...)
It seems that is the key different the creators of the tool are taking into account.
The difference is most were absorbed into The Melting Pot™ while people from Africa were never really allowed to assimilate fully.
Really, the core problem is centralization. If your website were a pub, you'd tell people to take it outside or 86 them.
Things can't work like that when most people will only patronize a small handful of enormous pubs.