Hacker News Reader

Comment by gregsadetsky | Hacker News Reader

Explanation from their Discord (more info/links below)

“Earlier tonight, we started having an outage using OpenAI’s GPT-3 Davinci model, which caused the show to exhibit errant behaviors (you may have seen empty rooms cycling through). OpenAI has a less sophisticated model, Curie, that was the predecessor to Davinci. When davinci started failing, we switched over to Curie to try to keep the show running without any downtime. The switch to Curie was what resulted in the inappropriate text being generated.”

——

A bit more detail:

https://www.vice.com/en/article/y3pymx/ai-generated-seinfeld...

And screenshots from their Discord with an explanation (this is quoted in the article above)

https://twitter.com/ryanbrianjones/status/162254854589825024...

View on HN

A seemingly innocuous engineering change led to some drastic consequences.

Imagine the other more critical production applications who have made the same reasonable engineering decision.

I dunno about that description. They essentially put a different brain in their guy’s body and he started acting different. One could consider that expected behavior.

This sounds logical, though my brain has always stored Curie as just like a Davinci divided by 2. They're right next two each other in the sandbox drop-down, it _feels_ swappable, in an innocuous way, or at least it did to me until this point.

I challenge your interpretation of a joke generator making a joke as "drastic consequences." Twitch has arcane rules about what can be streamed. As a streamer you have to tip-toe around topics (or jokes) regarding specific protected classes. This is a matter of arbitrary lines in the sand.

For a joke generator, making an offensive joke that gets the generator banned from the primary source it uses to reach users seems pretty high up their on the range of negative possible outcomes.

That's the point of the comment, if you were applying AI in more serious scenarios, would this be an AI confidently advising a doctor to overdose a patient, or advising a judge to incarcerate someone based on racist inferences from the set of people currently incarcerated?

> For a joke generator

I'm confused, I thought it was supposed to be generating Seinfeld episodes, not jokes?

Seinfeld episodes usually open and close with Jerry doing standup in a club, a format that Nothing, Forever emulates.

Or mental health: https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgp...

Am I wrong in thinking that getting your feelings hurt by an AI making an offensive joke is the equivalent of thinking you've been doxxed when a random number generator spits out your phone number?

This is a great analogy!

  $ seq -w 9999999

Isn't the purpose of machine learning to learn something, not nothing/randomness?

The point is that whether it's an RNG or language model or just a program printing a list of numbers like I posted, it's just a state machine spitting stuff out, it has no intention. So it's not really productive to get mad at it just because it produced something you don't like. The language model is spitting out random stuff based on the representation it has learned. It doesn't know what it means, nobody is making any kind of value judgement that results in it saying stuff, so why get offended by it?

Edit:

  Because if our adversaries are the products of inanimate and unthinking evolution, we cannot regard the problem in terms of revenge or payback... that would be no different than whipping the ocean for having sunk a ship and drowned its sailors 

  - Stanislaw Lem, The Invincible

I think it's fair to say that the anger people might feel is probably directed at the creators/owners of the AI, rather than the AI itself

That's like saying a book has no intention. It's just a bunch of ink and paper - it's completely neutral!

However, the AI may not have been trained in a neutral way, and the person using it may not be using it in a neutral way.

If you create an AI product, you are responsible for its output. People aren't mad at the model itself, they are mad at its creators. Why did they create an AI which ends up insulting people and breaking the ToS of the platform it is on?

If I understood you correctly your point is AI should be treated as RNG.

Here are three models: * A language model that outputs jokes, * A self-driving car model that outputs driving instructions, * An autonomous combat drone model that outputs engagement targets.

Where do you draw the line? Where do you start to "get offended"?

Just saw your Lem quote: Isn't an AI exactly not "unthinking"? That's the whole purpose of machine learning: to "learn", to recognise patterns, to abstract away, no?

The output has an element of randomness however. Regenerating the output will give you a different, albeit similar, answer.

In that case, instead of Seinfeld, AI should make a show about something.

Yes because you are treating a sequence of words the same as a pseudorandom generator.

The former has a probability distribution of sequences reflective of the dataset and context whereas the latter is pure random chance assuming input seed is random.

It's just a program, though. It has no bigotry. You don't punish a machine, you correct it.

This ban doesn't punish the machine; it doesn't care whether or not it's banned from Twitch. It does two things:

* Makes sure the people controlling the machine are incentivized to correct it (not that I think they needed it, but what I think is irrelevant)

* Ensures that content that Twitch management doesn't want on their platform, and which is in violation of their TOS, is removed immediately, regardless of the thoughts/actions of the people controlling the machine.

Yeah, but you also don't necessarily allow it to continue to broadcast while exhibiting errant behaviour. Your interpretation of this as a "punishment" of the machine isn't necessarily correct, nor the only perspective that's ultimately important here.

As far as Twitch is concerned, there's a channel that's breaking TOS, repeatedly. They're not "punishing" the owner of that channel, they're just enforcing the TOS, something the channel owner agreed to when creating their channel.

OpenAI (which powers this Seinfeld generator) is putting significant resources into preventing it from creating unsafe and antisocial material. Until that is prefected, outside entities like Twitch can use traditional moderation tools, such as the 14 day ban issued in this case.

As dumb as twitch moderation can be, the issue is that you can't possibly task twitch moderation with knowing to which degree an offensive joke is relevant or not. If an AI is set up to spout something-phobic stuff 24/7 surely they are going to ban that, rather than wondering where is the degree of offensiveness that warrants a ban, they just ban when ToS is breached.

At least that's what would make sense, and Twitch moderation is far, FAR from making sense in general, but in this one thing they might be reasonable.

It sounds more like a dumb AI bot was banned by another dumber AI bot.

Part of Twitch is the social aspect and that includes a process where clips are created and shared from source material. As part of the ban those clips are no longer available because people can share those clips for malicious reasons. Sharing the clips can quickly cross from trolling to targeted harassment. Was the joke super transphobic? Not really. Would getting spammed with hundreds of unique users sending you the clip become transphobic? Yes.

A clip of the joke from the banned channel:

https://clips.twitch.tv/CalmFrailPlumageShazBotstix-ITcqL0Hh...

Getting banned feels pretty drastic, especially for the amount of time they put into this project to just have it wiped from Twitch. But perhaps it's not a perma ban?

And Twitch's rules only seem as arcane as YouTube and Facebook too. Maybe not Twitter.

Drastic is a bit strong. A cartoon character on an infinite streaming tv experiment made a bad joke (which appears to be more of a bad meta-joke). The fact that the stakes are so low is presumably why they changed the model. If it was flying an airplane or something that mattered, they probably would have taken more care. The change they made had a completely acceptable risk / consequence tradeoff

Drastic is relative to the expected impact of the application. There isn't much worse that can possibly happen to a bot-generated stream than having the stream taken down by the powers-that-be.

This is like saying that having something taken down by a mod on reddit is a "drastic consequence". I mean, sure...the content is removed, but in most cases, the reason for the action is subjective and somewhat arbitrary.

To me, this is less a commentary on AI than on the absurd sensitivities of community moderation.

"Website makes moderation decision some people disagree with" is a lot less novel or interesting.

Again, the drasticness of a consequence is defined by the system you're trying to build for.

If you build a bot to automate your streaming so you can have 100% stream uptime and you end up with 0% uptime, that's because your product failed drastically. (Keeping in mind, as arbitrary and subjective the rules of Twitch are, the vast majority of human streamers have never been banned for arbitrary moderation)

For many people Twitch streams are their primary source of income, this kind of ban can have quite a lot of impact. And then besides the ban there is the chilling effect because no one wants to skirt the rules and risk a ban. And that is what allows social justice warriors to dictate what others can make jokes about.

Does Hacker News have such a chilling effect because it also has rules about what you can and can not post?

Hacker News is not a generic entertainment forum. Twitch is applying a chilling effect to comedians.

> If it was flying an airplane or something that mattered, they probably would have taken more care.

I think the usual response to comments like this is: "Oh you sweet summer child".

Not that you're technically wrong, but from years of headlines it's often enough that even if extra care is taken, it's not always sufficient.

> innocuous engineering change

> reasonable engineering decision

They swapped out the AI model. That's kind of a big deal.

In my mind, if I look at a drop-down and "model D" is down but "model C" is up and I need the API running NOW, I'm going to change the option to the one that's working.

That being said, every company I've been at with a production outage has gone into "fix it and ask questions later" mode. Maybe other companies have more process?

But when you put it like you said in your comment, it does feel more weighty

In contrast the (non-AI) companies I've worked with have a rigorous change release process which includes impact assessments, back out plans, getting multiple sign offs including QA testing and post-release testing, etc.

I mean specifically for a production outage fix. For planned changes, most of the companies I've worked at have similar release processes

> A seemingly innocuous engineering change led to some drastic consequences.

If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-...

tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.

I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement.

Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.

I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter

Is changing the source of data really that innocuous? I find it not that surprising that the output changed with the input

Especially considering this is the description for the model they left:

> higher quality, longer output and better instruction-following

and this is what they moved to:

> Very capable, but faster and lower cost than Davinci.

If nothing else, the adage 'you get what you pay for' works.

ref: https://platform.openai.com/docs/models

I've only been through a couple production outages, but at those companies, usually we'd get things patched and ask questions later. I could definitely imagine switching to another model that is made to sound like a smaller version of Davinci, at least from how it's been stored in my mind up until now.

I've been there quite a bit, and it's pretty true! It's best to worry about how the ship started sinking after you've saved it :p

In that situation I still try to exercise my options a bit...

Seeing that this new model is implicitly worse at following directions, knowing that 'production' (this is a Twitch channel) depends on obeying a certain set of rules... switching to it may be ill-advised.

I don't want to be seen as overly judgmental - the Twitch channel comment works both ways; derogatory/supportive. I get why one wouldn't really foresee this or even care to. Then I wonder, why not leave it down for a bit? The hype train is fickle but it's not that precious, either.

They've been developing this on-the-fly since it gained popularity (and profitability). Basically, they ran fast and broke things, like any good startup.

I'm nearly 100% certain that the scripts are generated through prompt engineering, with a random prompt (e.g. tell a joke prompt, talk about a new restaurant prompt) being selected for the scene.

From what I can gather they first used the older, cheaper GPT-3 models, only upgrading to davinci-003 when it was profitable. The older GPT-3 models proved fine and didn't generate edgy content for the several months they were up and running.

But I think the change that broke the camels back was they added a "2006 Laugh Factory incident with edgy content" prompt and only tested it on the davinci-003 model - the new models having been wiped clean of antisocial training data, while the older smaller models still having contentious content encoded in the model.

So, davinci-003 did fine producing "politically aligned" text with the "edgy" prompt because it's "cleaned", but when the openai API for davinci went down the fallback was curie. The older "unclean" curie model combined with an edgy prompt inevitably caused what we saw here.

My takeaway is that the more we turn to larger and larger models (developed by third parties) that obscure their underlying nuances, the more risky swapping models in a particular domain becomes.

Cue MRE/MLOps job listings.

I like how the most "additional quoted" and "explained" reason has absolutely no merit because there's no automatic fallback between models. It only falls back to older model if you program it to.

This is why, like with every live show, you should have a delay and moderation so you can intervene before unwanted things are broadcasted and get you into trouble.

So, I watched the clip, and honestly that sounds more Norm MacDonald than Jerry Seinfeld.

Definitely not Norm. Norm would (and did!) say he loved transgender people, and would leave ambiguous whether he was being candid. (Or he'd pretend to struggle with his love for transgenderism versus his distaste for "psychosexual sadists who slay people" - not looking it up but pretty sure that's an accurate quote. The joke of course being that he is confusing two different categories of people in a shocking way, playing up his clueless character.)

He liked bombing, but only with jokes he believed in. He wouldn't tell a meta-joke like this with such a lack of punchline.

Is there any info on how long the ban is for? I assume a week, but forsen’s was a month.

14 days according to the screenshots posted by the GP

0 days if you make a new account

This makes sense because most of the political ChatGPT headlines I see are it refusing to say nice things about republicans or crashing because it's unable to say anything discriminatory. Using a different model here makes sense

What’s “errant” about it? Lots of transphobic people around, if anything I’d say the AI was closer to “real life” than lots of other AI systems.

Would it have been transphobic if the sentence had been prefixed with 'some people say...'

At a certain level AI trained on inputs is just a big 'some people say' box.

People saying transphobic stuff are indeed transphobic.

Not looking for a flame war, and I know this is not an acceptable take, but y'all need to learn the difference between conversations about a topic and conversations supporting a topic.

That assumes an objective ruler who defines what is transphobic.

Reply on news.ycombinator.com