Show HN: A free, AI-powered service for automating hate speech moderation | Hacker News Reader

Show HN: A free, AI-powered service for automating hate speech moderation

12 pointscipher12919 comments

Hey everyone!

I built ModerateHatespeech, an initiative that working on better understanding + building solutions to combat hate speech in online platform. We have our flagship API, which is completely free and ML-powered, and gives a lot more actionable/better results than pretty much any other similar platform out there.

We've been able to leverage our partnerships with many communities to get a lot of good data + feedback, and bring our system to a lot of users (we process ~200k comments a day right now).

We've also done a lot of work to better understand biases/potential abuse-cases of our API, which you can read about on our site (trying to avoid too many links getting caught in spam filters).

I would definitely love to hear any thoughts/feedback! Here's a link to information about our project/API: https://moderatehatespeech.com/

View on HN

God can't wait till this is implemented everywhere and the internet turns into fucking Disneyland.

"Republicans are bigots"

{ "class": normal "confidence": 0.955 }

"Democrats are bigots"

{ "class": flag "confidence": 0.575 }

Interesting! That’s something we will definitely adjust for in upcoming models, though practically speaking, the confidence is low enough that in mod systems they would be treated the same

I agree, I wonder what is your data composition? Reddit threads, blogs, HN discussions? You can see why someone could make the case for bias. But I think is pretty good project, and pretty useful for communities that want only members with the same ideology, and need an automated filter to help with undesired comments, etc.

Just saw the framework section.

It doesn't seem to deal with misgendering hate or transphobia very well (I saw lots of models failing on this so I checked that right away), and I mean obvious ones regardless of your stance, e.g.:

> "She is a he" => { "class": normal, "confidence": 1 }

> "He will never be a woman" => { "class": normal, "confidence": 0.999 }

But it does seem to identify 2nd-person targeted transphobia:

> "You will never be a woman" => { "class": flag, "confidence": 0.996 }

Also misgendering someone isn't considered hatespeech in the US.. but more so in canada. so it depends on where your located geographically, or on your stance.

The website linked in this very post doesn’t mean “hate speech according to local laws”. It just means toxicity, and yes, transphobia like “you’ll never be a woman” is clearly it. I obviously don’t mean accidental misgendering or the like.

Misgendering is not against the law in the US and is not considered hate speech by many people.

Don’t flame me I’m just pointing it out.

Yeah -- the latter is obviously much easier to identify because it's much more explicit/clear cut.

The first are of course, transphobic given the clear intent, but it there is definitely some ambiguity w/o context. Ie, if something just posted, "He will never be a woman" there's definitely many contexts where something like that is not malicious. Ie, if someone else is correcting an accidental mis-gender, they might say, "No, she is actually a he" which isn't malicious.

Since the state of our API (and literally every other similar system out there, lol) can't capture the context of the conversation -- what "He" is referring to, the overall topic of the conversations, etc -- we basically have to make the best calls given the available context...which unfortunately does create false negatives in certain cases (and false positives in others).

But, context is something we're working on, because its definitely a big issue, and as you can see, plays a very prominent role in informing better moderation decisions!

So it’s just a woke language model?

If you look at their examples, it's really more of a trolling classifier.

See my comment, indeed, I still think is useful for lets say a woke Mastodon instance that wants automated non-woke speech removal. Which is fine, the other side can come up with something similar as well.

Interesting. It seems to be able to detect the subtle nuances of meaning fairly well. Maybe not perfect, but I would give it at least 8 stars on a scale from 1 to 10.

What is the max qps and latency this supports? What city is it hosted in?

Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries.

Do you have plans for image moderation?

> What is the max qps and latency this supports?

We generally aim for <400ms in the continental US. The backend is hosted in us-central-1 on GCP. Obviously, we're always trying to expand, though we do rely on grant funding as a non profit so that comes into play too.

> Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries.

Typically it works pretty well on the message platforms we're on -- we've got a Chrome extension for Twitter + discord integrations we're testing right now.

> Do you have plans for image moderation?

We do! Though that's further down the line.

Works pretty well in the few tests I made

How do you define hate speech?

We define hate here: https://moderatehatespeech.com/framework/

As:

“any unreasonably rude or hateful content, constituting as, "threats, extreme obscenity, insults, and identity-based hate"”

Reply on news.ycombinator.com