Hacker News Reader

Comment by pwython | Hacker News Reader

I've had dreams of building something in the image sharing or social platform realm, but stopped short of planning because of obvious content moderation responsibilities. This seems to be a realistic, cost effective solution to that one piece of the puzzle.

View on HN

OpenAI's moderation API is multi-modal and free with no strings attached in a way that truly boggles the mind.

I've put easily over a billion requests (>$100,000 by typical moderation API pricing) through it over the last few years for $0.

I think it's a severely underappreciated offering, but I also don't bother pushing it too hard because who knows when the party will end lol. Strikes me as something that's only stuck around because no one's abusing it.

Yes it does look like a good solution. But when I imagine actually using a guardrail for a product, this model only outputs yes/no probabilities. There is no reasoning trace why it was rejected. Users or even developers would have no idea why a prompt was classified yes or no. I really like this release but I feel like I need something more to use it as a guardrail in production.

I think IRL in the “rejection” case they don’t want to tell the user exactly why, since the user may be malicious and use it to try to evade the block. And for use in moderating UGC, well, most platforms don’t take seriously the idea that they need to answer to their users. Only their advertisers.

In the case of wondering why a bad thing got through, well, I think that’s why they just set these to the most pro-censorship level they can, to make that highly unlikely.

Rejection reasoning is also a liability and can be extremely legally risky. If you're the company, you don't want a user to win a lawsuit against you just because a judge disagreed with the exact reason you banned someone.

I am not sure how reliable it is in the real world. Also, in terms of liability, I don’t know how effective it would be to satisfy various regulations compared to a human moderator team.

I hear ya, but one could set different operating thresholds: auto-approve low-risk posts, hold ambiguous posts for review, and automatically reject very high-confidence violations. So HITL for sure, but MUCH less H in the L.

Human-somewhere-nearish-the-loop!

Reply on news.ycombinator.com