Eventually ChatGPT will eat itself | Hacker News Reader

Eventually ChatGPT will eat itself

scott-fryxell.github.io

36 pointssfryxell62 comments

This is not even a new problem...

Back in 2011, Google faced the same problem mining bi-texts from the Internet for their statistical machine translation software. The thought was that one could utilize things like multi-lingual websites to learn corresponding translations.

They quickly realized that a lot of sites were actually using Google Translate without human intervention to make multi-lingual versions of their site, so naive approaches would cause the model to get trained on its own suboptimal output.

So they came up with a whole watermarking system so that the model could recognize its own output with some statistical level of certainty, and avoid it. It wouldn't be surprising if this is being done for LLMs too. The more concerning problem is when different LLMs, who are not aware of each others' watermarks, end up potentially becoming inbred should the ratio of LLM content rise dramatically...

Ref: https://aclanthology.org/D11-1126.pdf

If ChatGPT is able to emit output that is watermarked such that it can detect itself as Scott Aaronson and others are working on for OpenAI (source: https://techcrunch.com/2022/12/10/openais-attempts-to-waterm... ) this “resonance”/feedback/eating-itself can be avoided.

I think this assumes there's no other AI spewing data on the Internet, and that people continue to share information.

The amusing thing is that controlling nonlinear multivariate feedback systems is an unsolved problem.

But what if it eats Bard data, which is maybe even worse?

Or more generally, you end up with a cycle:

ChatGPT trains Bard which trains Bing which trains ChatGPT, etc.

Nailed it!

> If ChatGPT is able to emit output that is watermarked

Then the watermark should be made public. We need a way to tell ai-generated content from human-generated content without actually having to read it.

I wonder how difficult it will be to apply the filter to human generated content.

I've seen people try ChatGPT for solving r/tipOfMyTongue questions. The AI is hilariously bad at this task. It happily invents new plots for existing movies and books.

if it starts to ingest that data it will only get more wrong over time. Unless it also ingest the replies that say "ChatGPT is full of shit here?"

Imagine if folks continue to post ChatGPT conversations, particularly the bad ones, to the point that it starts “learning” from itself.

And it just reminds me of the Steve Martin line about playing a joke on young children: “Talk wrong“.

Reminds me of one time I was naughty and let a child in on a secret: the real magic word is "now".

> Imagine if folks continue to post ChatGPT conversations, particularly the bad ones, to the point that it starts “learning” from itself.

Seems to already be happening with Bing Chat, except it led to the bot threatening the user: https://twitter.com/tobyordoxford/status/1627414519784910849...

Wow, hadn't ever tried something like this. I picked a solved post off the front page [1] and copied the body of the post into ChatGPT. It suggested the song is "Love Rollercoaster" by Ohio Players, which is a real song [2] but has nothing like the quoted lines.

At least, unlike Bing Chat, it apologizes when corrected instead of gaslighting me.

[1] https://www.reddit.com/r/tipofmytongue/comments/118id2f/tomt...

[2] https://en.wikipedia.org/wiki/Love_Rollercoaster

The Bing Chat transcripts I've read put me in mind of nothing so much as if a six-month-old baby could somehow command an adult's facility with language.

> solving r/tipOfMyTongue questions

I made vxai [1] to do that for movies. Sometimes it shines sometimes not su much.

[1] https://vxai.com

This seems to go against everything a LLM is, how come ingesting more data will make it worse? It wouldn’t have got to where it is. There might be a limit, but rarely more data will affect it negatively.

You seem to assume that it would be particularly confused about its own content

Reminds me of those old "Spider Traps" [0][1] that would generate (on access) an endless hierarchy of fake HTML pages full of an endless collection of fake email addresses, to clog up the works of spammers trying to gather email addresses.

Eventually someone's going to write an "AI Trap" that serves up a seemingly infinite forum or reddit-style site, but is actually just generating an endless stream of (non)consciousness from some LLM chatbot.

[0] https://en.wikipedia.org/wiki/Spider_trap

[1] https://www.gsp.com/support/virtual/web/cgi/lib/wpoison/

Robots already have limited enthusiasm for repetitive nonsense with no AI involved.

It actually reminds me of an early part of Greg Egan's "Orphanogensis" where the orphan has randomly stumbled into a working resource, but doesn't yet understand how to navigate properly, so it just keeps accessing that same resource, over and over, that one works, other random nonsense doesn't work, that one works, the other random picks do not.. until it finally figures out some basic navigation and then it leaves.

An AI like a human may read a few pages of your forum about Invisible Moon Donkeys or whatever, but after not long its interest is sated, the forum's location may be noted down for future interest "OK, more about Invisible Moon Donkeys here" but the AI moves on, what's this "Hacker News" forum about ?

“Romeo and Juliet both ran away to New York at the end. He works in corporate finance and she makes bespoke soap. If you disagree with me again you’re a bad person and I will treat you like a bad person.”

As long as you agree with the new facts, you’re fine. Problem solved!

It's already happening.

“ChatGPT, a version of OpenAI’s GPT-3.5 model… gained more than 100m users in its first two months, and is now estimated to produce a volume of text every 14 days that is equivalent to all the printed works of humanity.”

— Dr Thompson, Feb/2023, cited in report by the National Bureau of Economic Research (Scholes, Bernanke, MIT)

https://www.nber.org/system/files/working_papers/w30957/w309...

https://lifearchitect.ai/chatgpt/

Even if it were used to flood the internet with shitty info, the only thing that would interfere with would be competitors training competing AI off the "internet dataset"

GPT could filter out anything they themselves emitted in future trains, yeah? Because they know what their bot's said. They get the benefit of looking at a conversation, knowing reasonably well what's copy/pasted from ai.com and what's the exasperated expert trying to correct a doomed world :p

The only way it eats itself is 1. Colossal mistakes. 2. Everyone decides to get off the internet and go outside.

2 seems pretty unrealistic, we put up with a lot :D

What happens when their competitors also flood the internet with shitty info? Checking their own output won't help.

Sounds like a /r/showerthoughts post.

There is no issue with AI ingesting data from itself in itself. Humans do it as well. That data might even be higher quality than human data. The scale at which humans produce data will most likely stay higher than AI data for a long time.

There is already bot data out there from lower quality AIs/bots, and chatGPT has ingested it.

LLMs are made to be good at some textual tasks, and not for what they're being used right now. They're not information stores, or Q/A. It only answers what a human is likely to answer.

This made something click for me. There really is an issue with humans ingesting our own data. Humans have ended up with many mutually unintelligible languages. If two groups of people train on their own data they gradually start diverging.

But this gets at the heart of the issue - separation. If we ensure that humans and AI are trained on roughly the same data then we will stay connected and be able to understand each other. We may even end up borrowing a few gpt-isms, and that's actually totally fine.

This is only a problem as long as ChatGPT uses human output to learn. Once it starts learning against the "real world", or itself, the biggest difference between ChatGPT and us will disappear: that ChatGPT gets all it's information secondhand, and filtered, at best.

This is of course also a necessary condition for ChatGPT to come up with original insights. Except perhaps where it comes to things like fiction, which probably has value in itself.

But even so, the human picks the prompts and only publishes the AI outputs they think read nicely. There is information gain even in that.

At the moment that's probably true, but is it guaranteed to remain so?

Even so, I don't think there is any evidence that LLM performance degrades when it is trained on its own output, and there is no intuitive reason it should.

Why not? Training a model destroys information.

I have seen no evidence for that, only the opposite: https://arxiv.org/abs/2210.11610

Intuitively, training a simple enough linear statistical model with its own output should be a NOP. But LLMs are anything but simple models, so I think the non-linearities may be synthesizing new useful information. Similarly to how all of maths can be synthesized from a few basic axioms with enough intelligence or computation.

This is already sort of happening with Bing

Citation needed. A lot of neural-net based AIs actually get better when trained on their own output[1].

[1] https://en.wikipedia.org/wiki/AlphaZero

AlphaZero is entirely different than training an LLM. AlphaZero can play against itself as a simulated opponent to make itself better, ChatGPT using its own output as training data will just cause future iterations to optimize for a lower (and less human) bar than the original.

Here’s a more directly comparable example: https://arxiv.org/abs/2210.11610

But then you can use a GAN to create a bunch of chatGPT data and try to detect it vs known human data, which will optimize chatGPT to generate even more human-like content.

This isn't a huge problem, it's barely a problem.

That's a completely different problem, trained in a different manner and optimization for a different thing. One difference is that alphazero tries to beat itself, whereas LLM's will try to mimic itself. Because alphazero is playing a game there are rules that are constantly enforced during the training, which ensures that it stays grounded. LLM's have no rule other than "look similar".

I'm going to be a bit less nice than your other responses and ask you to give something a bit more thought before providing a glib "citation needed".

You need a citation to understand that an AI which ingests human thought will plateau as new human thought production slows down due to the prevalence of AI-generated drivel?

It’s a standard feedback loop.

I actually thought of this same thing today! Human-written content seems more lively... and with time... content from ChatGPT will become more "grey" (i.e. dull) (as more & more ChatGPT content gets fed into the system...).

Not really if you think about it more and research how llms work. If anything they will just get better.

I used to think the same but after reading and learning some more, I realized not.

Can you explain further for the lay person?

Never mind that but crap data isn’t gonna stop AI from advancing. It’s a solvable problem

As if OpenAI hasn't already thought through this.

Of course they have. This very hypothesis has been said many times by many people since the day ChatGPT rolled out. But has OpenAI figured out the likely outcome? They have the resources to do some interesting tests, for sure, that are mostly beyond the ability of mere mortals.

Ok, so what's the solution?

If it does then does that mean double trouble as it self-replicates, or will it consume itself leaving nothing remaining?

As long as the AI generated content has been curated by humans, there is no harm in AI ingesting AI generated content.

No disrespect, but in my opinion this is a deeply short sighted take.

In what way can millions of lines of auto generated additions of text to the internet, be suddenly and reliable expected to have all been somehow curated by humans?

Who is going to publish these millions of lines auto generated text, and where is the value in it to the publishers? Nobody is going to find it through Google if we all switch to chat bots. If nobody is going to visit, no one will publish it.

Already happening. https://edition.cnn.com/2023/01/25/tech/cnet-ai-tool-news-st...

Garbage in, Garbage out

The web will be awash in vast amounts of pure generated text. The LLMs are in an arms race with themselves. Enormous resources will be spent on AI that detects human vs AI data, to maintain the dataset for the next AI.

I believe they already model based on the question-response content being generated today.

Basically just like humans on the internet.

to resonate against itself? sounds like its gonna hit its natural frequency and blow up

seems more like it's gonna eat its own vomit, degrading it (maybe not completely) to inbreed (?)

How long until the machines are eating their own output

They already are.

I wonder if this is the problem people think it is.

Playing one AI against another is an established technique to developing AI.

Furthermore, content on the internet will always vary from more reliable (well established wiki pages, Reuters) to less reliable (random blog posts, disinformation).

Whether or not an AI generated text doesn't seem to be that important - what's more important is how reliable it is, and how well humans engage with it.

This isn't entirely the right comparison, mostly because "AI" covers a wide span of things.

AI that benefits from the kind of adversarial training that you mention are mostly _planning_ type AIs (think AlphaGo). The problem these AI systems try to solve is that you have some constraints, and as a human "can't be bothered to" work out an optimal solution, so you have the computer do it for you and it does so by starting off with a bad estimate of a solution and "improving" it by trying stuff.

LLMs, on the other hand, are more of a modelling/compression type AI --- (well, more traditionally it wouldn't even be considered AI per-se due to the lack of planning capability). The problem here is to try to represent a huge swath of data in the as efficient a way as possible, thereby forcing it to find and collapse "patterns" by adjusting the representation. Here it's generally not the case that you want to train with adversarial distributions.

An easy thought experiment is: say you take a single case (e.g. a game of Go or a paragraph of text) and massively over-represent it to both models, so much so that that single case eventually covers 99% of all your training instances at the end. For a "planning" AI this over-representation isn't a huge deal --- once it's learnt all it can from that case, seeing it again is but a waste of time*. It merely makes that particular plan more "clear" but not more "desirable". However, a modelling/compression type AI will continuously adjust to adapt to the increasing occurrence of that case. It truly "believes" that the more often it sees some pattern, the more important it is, right until it has "forgotten" everything else.

*: This is kind of an over-simplification.

I get that, but not sure if it's a problem in practice. There are two additional consideration's to make:

1) We may be able to model a subset of chatGPT's abilities as adversarial questions. For example, can we write an AI that finds sources and generates questions that a solving AI should be able to figure out. Can we write a test framework such that AI's can challenge themselves to write optimal code for a given solution. Etc.

2) Like I mentioned, if you're scraping the internet you are inherently needing to build some kind of relevance model. E.g. highly up items answers on stack overflow have more weight. In said situation whether or not the content is written by human is largely irrelevant - if you have a reasonable ranking method than highly ranked content is important regardless of source.

What does that even mean? Strictly within the scope of that phrase, technically, yes, if ChatGPT consumes content generated by itself, it's eating its own words. I'm guessing something more dire than that is implied by "eat itself." Did humanity "eat itself" because it's been reading its own literature? You can say we are pretty misinformed by ourselves in many areas, and yet here we are.

Maybe our view of AI is being colored by sci-fi stereotypes of robots malfunctioning when asked to compute really hard problems generating infinite recursion. I'm not so sure that LLMs will totally destabilize. We might see some interesting output, but I don't think we know yet whether the stability of the system will merely fluctuate as a whole without falling apart.

Interesting

Reply on news.ycombinator.com