Hacker News Reader

Comment by Bayart | Hacker News Reader

Question to people interested in designing those types of models : what does it matter if the output is "toxic" ? People are toxic ! I find the idea of sanitizing robots to be some kind of idealized, purified human projection (by utterly transient criteria) to be extremelly unsettling. The current sanitization of public discourse is already terrifying. Now you want that crap to be AI-powered ?!

View on HN

Even worse, filtering the data in this way makes the model biased, i.e., not a useful model.

Imagine for example that someone created a model of human behavior...but filtered out behavior considered "toxic" (whatever that means). Then someone else would naively create for example a simulation of the reaction of a large group of people to some expected events. Now there's a chance that if these events occurred, that mass of people would actually react very differently...because the model was biased! I.e., it did not approximate the average reaction since reactions in some directions were artificially clamped. So now any decisions based on that model would be worthless.

That’s making a lot of assumptions. This isn’t the sort of model that you could base such a simulation on. It’s glorified autocomplete. (Impressive for what it is, but unreliable.)

I'm speaking in generic terms, of course. All models should be at the very least unbiased, though -- that's the least demand you can make of them (and generally an easy one to check, compared to other properties, such as having minimum variance of all possible models and such.)

I sort of agree in principle but I’m not sure what unbiased means for a language model or how you would measure it, so this doesn’t sound all that easy to me.

Right, especially since what's considered "toxic" is a moving needle (that quote of the General de Gaulle that Zemmour like to use for instance was perfectly fine at the time it was said and still is for a lot of people) and is also not uniform across the world. In particular sexual jokes or double entendre are perfectly fine in French most of time while they're often considered harassment in the US culture.

We could think of language models as public figures and developers and users as supporters. Every public figure has heard nasty language: it's up to you to decide whether to support a public figure who goes around repeating said language.

It matters if people figure out how to make your model output inappropriate things and start sharing screenshots on twitter. What if you use a GPT model as part of a chat bot? Then you definitely don't want to risk any sexual or discriminatory output for example. If that was a person talking, they would be fired.

There is no bot to fire here though, as it's not employed.

Reply on news.ycombinator.com