Researchers warn of ‘model collapse’ as AI trains on AI-generated content | Hacker News Reader

Researchers warn of ‘model collapse’ as AI trains on AI-generated content

venturebeat.com

79 pointsbelter53 comments

This is what happens when your objective function is to generate plausible or well-aligned next-word predictions - the model will converge on agreeable nonsense that sounds credible but is increasingly divorced from the truth. RLHF is a lousy patch to fix this problem. The real solution to grounding LLMs is by training it on hard-to-solve but easy-to-verify (i.e., NP-complete / hard-on-average problems), then formally verifying the generated outputs. While this may be difficult for prose, by including a large amount of training data containing formal reasoning tasks and programmatic text, this puts a strong bias towards correctness.

Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obvious problem with LLMs right now.

The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks.

The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in certain logical theories. Although there is a growing evidence that Transformers and GNNs have foundational limits to expressivity [3, 4, 5], if the scaling crowd could show strong generalization results on these sorts of problems they would have the last laugh...

[1]: https://arxiv.org/pdf/2210.10749.pdf

[2]: https://arxiv.org/pdf/2305.18654.pdf

[3]: https://arxiv.org/pdf/2106.16213.pdf

[4]: https://arxiv.org/pdf/2301.10743.pdf

[5]: https://arxiv.org/pdf/2303.04613.pdf

> The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set.

I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one...

The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).

It seems like you will have to actually instruct the model like a human pupil with explicit feedback on what is correct and why.

An LLM is Always making things up - that’s what it Does. It is a corpus of word (token) probabilities and a function to generate probabilistic paths through them from a starting point. It is always hallucinating, it just often hallucinating things that also happen to exist.

You've constructed a very elegant frame here that says basically nothing at all.

Isn't this exactly one of the problems that GPT4 has been aiming to solve? It'll be interesting to see if this is the tyranny of AI -- whether degradation of NLMs is an increasingly insurmountable barrier to good performance or whether it's something that can be massaged out with better verification.

You mean like brute force password search or prime factorization ? That won’t help. There is no free lunch in compression, search, or learning from data.

This makes me laugh. Reddit is trying to weather a user blackout because it thinks it will cash in on selling user-written content as AI training data. But Reddit itself will be filled with AI spam from now on, so the data will be worthless for the purpose they want to sell it for. Serves them right.

Will this still happen when humans are in the loop?

Isn't this kind of like running an evolutionary algorithm on the models?

- AI generates 20 images

- A human picks out the best 2 images to publish to the internet.

- Eventually the 2 best images end up in the training data of a new model, while the 18 bad images get discarded.

- Successive generations of models bias more towards the better images. This might get rid of some of the defects that plague the worse images.

Humans end up being the fitness function and even the results from different models ends up mixed together!

I'm not sure that this would be a big problem for images, because there usually isn't an emphasis on correctness and validity of information. It seems more of a problem for LLMs.

Are humans still in the loop?

1. AI (say ChatGPT or whatever LLM you like) makes blogspam

2. AI (Google Pagerank etc.) fooled into high-ranking it

3. AI (unnamed scammer) searches for a suitable comment form to post spam on, pretends to be real human as it does so

4. AI (Google Ad network or similar) selects best bid for an advert to show to the fake-user in #3

4. a. AI (Amazon, eBay, etc) has meanwhile been placing bids:

dictionary.filter($0.isNoun)).map { "\($0): Buy it cheap on Amazon!" }

5. AI (products scammers) look at "best" opportunities on Amazon, create fake listings to match each noun in 4. a.

6. AI (same scammers) upvote the fake reviews for the fake products for the fake nouns advertised to fake users on fake blogs,

but the metrics… which are A/B tests… which is GOFAI (AKA applied statistics)… say it's good, so everyone keeps doing it.

I like how you are thinking and I wondered if the company operates as a monopoly or is in a very small market, people won’t have alternatives to pick from and so their demand inelasticity combined with AI will lead to bad choices about A/B tests. I assume this is already problematic without AI, as it seems many company’s don’t know what made them successful as they ascend and A/B tests lie to them. When their product is in hot demand they think every stinker is a rose by misinterpreting the noise.

still some humans in the click farm

> Will this still happen when humans are in the loop?

Yes it will. Unless the humans are generating new high fidelity content.

Even if humans do generate high fidelity content would it help when it constitutes a miniscule sliver of all new content?

in many fields, it'd be the largest sliver. perhaps, in general, it'd be fiction collapses and non-fiction grows?

In some senses that will be the greatest damage. We live our lives in the fictions we share, using them as touchstones for how we talk to each other. Generative AI is the genie we've let out of the bottle, and we keep making wishes.

The invalid presumption is "A human picks out the best 2 images". People will absolutely stop looking at these in any meaningful way (or even produce them in a direct-to-the-internet pipeline)

In the future, we will have air gapped facilities where people are paid to harvest their thoughts, similar to plasma "donation" ;)

In all serious, I've been talking with my colleague a lot about this.

Will be interesting to see if mining archives of "low background steel" data (anything created before chatGPT exploded) will become a industry.

Another interesting impact... older, more established tech suddenly has a non-trivial advantage over new, cutting edge competitors because LLM are well trained to work with the older tech.

Unnecessary. In my view, the immediate answer to this is to let AI train in the real world, for instance surveillance footage, microphones in public locations that will be able to hear and transcribe entire conversations with humans in the “wild”, environmental/satellite sensor data, AI controlled drones with cameras that the AI can “explore” with etc. Additionally, humans interfacing with the AI is also going to be a source of data especially if an AI can “interview” people.

If anything, I can see a future with a push to further erode privacy laws with the ultimate aim of getting more access to human data in the real world to train AI. I’m sure the smartphones in everyone’s pocket is a rich source of sensors and data…

If we can get to a point where AI can reliable seek out, filter/normalize and incorporate data from the real world into its model, that’s when it will probably start approaching the singularity.

This sounds like a horrible world to live in...

Maybe, but it would be very interesting to see an AI drone exploring every corner of the world (in a low/no impact way), categorizing species we’ve never discovered, finding patterns in environmental sensor data, making recommendations on which rivers are the biggest polluters and the best places to intercept the problem, determine the best way to optimize rain catches to increase groundwater etc.

The obvious flip side of which would be that authoritarian governments could use this to figure out how to crush dissent, carry out “efficient” genocides, pillage their forests etc.

This seems inevitable to me once we have cheap APIs with multi-modal abilities.

Surveillance is what AI will excel at and so that is what it will be used for.

Maybe we need to integrate the AI golems as persons into society, so they can learn and interact with us as individual persons, like us.

But remembering Battlestar Galactica, cutting the interconnectivity was the prominent defense against cylons.

My standard reaction to this is Batman’s city-wide sonar machine. (Not that I know anything about it)

Or 100% certified Humans willy offer themselves to interact with your AI. For a nice hourly fee of course. A Human has got grown embryos to feed..

I makes me want to build a library of books that aren't available in a digital format

We call this, “Reddit”.

There's a decent chunk of reddit activity that was bots even just a few years go. I doubt AI has made that problem any better

Garbage in, garbage out

The real moat openAI might have in the future is a “mostly human” dataset they managed to scrape before the generative AI frenzy

"The real moat openAI might have in the future is a “mostly human” dataset they managed to scrape before the generative AI frenzy"

How useful will this dataset be in 10 years, though?

Would ChatGPT 4 trained up to 2013 be useful now? Probably not so much...

This is one of those things which is plausible sounding but probably untrue. We are generating exponentially more data with every passing year, only a small fraction of which is going to be AI-generated.

Consider:

- you could put a microphone in a busy public place and do speech to text on everything you heard

- you could train on audio data from films etc. Yes some (tiny if the WGA and others have their way) fraction of the script will be AI generated but a lot won't be, actors will improv etc.

- There will still be a massive and ever-expanding amount of human-generated text to train on. For example we have the fast-growing niches of AI doomsterism and AI mindless optimism.

- You could hook it up to gemini to find out what the hipsters are thinking

- public domain books are a massive training corpus and every year a new vintage gets added. It's going to be quite some years before the AI-generated garbage tier gets through the copyright period and out into public domain.

- In certain cases AI can actually help to train AI models. Obviously we saw this in the reinforcement learning explosion around AlphaZero etc but we're starting to see it in language models with things like voicebox[1] being used to train other models, and Facebook's paper shows significant model improvements from training this way.

[1] https://about.fb.com/news/2023/06/introducing-voicebox-ai-fo...

Casey Newton proposed, partly tongue in cheek I think, that as LLMs advanced, it might spur investment in and a newfound appreciation for Journalism as way to provide fresh source material for the models

How would that investment generate a return, though?

"Google and OpenAI discussing agreements to pay over using content to train AI" - https://news.ycombinator.com/item?id=36368743

Especially given GPT-based news summarization services.

Paul Graham already made a tweet a few months ago addressing this issue, saying: “We can't all use AI. Someone has to generate the training data.”

You can find the related thread (603 comments) here https://news.ycombinator.com/item?id=35159447

The issue is because of the current nature of LLM. Basically an LLM is attempting to model its inputs but it tries to auto regularize/generalize, which means information loss. And then if you use that LLM to generate more content to train another LLM, once it is regularized/generalized it will have further information loss.

This isn't that different than re-compressing images over and over again. They are disconnected from reality and are just simulations of reality and with each iteration they become less rich and faithful.

AIs need to learn from the world or at least as close as possible to it, rather than from their own reduced representations of the world.

But the output of the LLM is trained on a synthesis of multiple inputs. Even if every input has information loss, new and novel combinations can be created. I imagine it’s the same for humans since we have lossy memory.

It doesn't matter if it is multiple inputs if they are just approximations of approximations of approximations. You need to engage in something that isn't a rough approximation, but something richer than your representation.

Humans have lossy memory but we constantly engage with the real world so it doesn't matter. The key here is we engage with the real world on a regular basis.

The key idea is that you can improve a low fidelity representation by training on a higher fidelity representation. But you can not get a higher fidelity representation from a low fidelity representation. Any inference is just making things up and likely to be wrong, so it is more like to just have less details.

The last 20 years have made it obvious humans seem pretty apt at collapsing reality models without AI.

This was my immediate first thought, too.

Arguably AI and large scale social networks have a lot in common (opportunities and problems).

Related: AI as the logical next step after “crowd sourcing”.

Pretty conclusive evidence that human creativity is the source of current "AI" power, no? Many people claim the plagiarism inherent in this product is no different from you paraphrasing & remixing material you've read elsewhere, published by other people. But no, humans don't degenerate when reading material produced by other humans. They improve upon it and use it to create things that are genuinely new.

Maybe this won't always be so for AI. But it's pretty hard to square an account of LLMs as more than fancy data compression when they exhibit the exact same tendencies as all compression protocols when run on their own outputs.

Humans are the batteries in the matrix.

No one knows because AI do and can generate useful training data, example being asked to generate the same image with added artifacts. But you have harmful ones like generating a repeating pattern or data that looks different but is very much the same.

When you say useful, what does useful actually mean? If ai is just a tool and humans are the beneficiaries you can kind of frame useful as useful in some way to humans. When the human is out of the loop completely how do you understand if you're going in tge right direction?

Hyperbolic article from writers that didn’t understand the papers.

The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets, which is obvious if you think about it. For instance, the GPT training set doubtless includes plenty of swearing and slurs, but its output does not (even if you prompt engineer your way to an example, they will still be less represented in the total output).

Same is true for the long tail. If I ask GPT to complete “oregano tastes like”, there are lots of fairly likely completions, but it’s probably never going to produce “tires”, even though exceptionally rare combos probably occur in the training data.

I’m not sure this is a bad thing. Maybe? Is there some productivity or moral imperative to leave the distribution of words/concepts the way it was before LLMs?

> The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets

Yes, and that seems pretty much unavoidable even without SFT and RLHF "filtering"... There's no obvious way of recreating the "long tail" distribution of the overall training set. Turning the sampling temperature up isn't the same since that'll affect the entire generated output, while a human might have chosen to pop a single spicy word choice into an otherwise boilerplate sentence.

One way to get more human-like variety of these models would be to condition them on specific writers with different styles, but of course in practice certain styles would be used more often than others and a corpora of machine-generated text would therefore not have same statistics as one of (more randomly chosen) human text, even if it did have more variety than would otherwise be the case.

Still, conditioning on a specific writing style might be a future direction. Would be useful for people wanting to generate text in their own personal style rather than just asking the LLM to adopt some generic persona.

Actually, AIs have been training on AI-generated content for a long time now. An example of this is student-teacher distillation.

It has to do with quality of the data, not that it is AI generated. The AI generation is depending on what we give it.

nonsense paper, nonsense idea.

Reply on news.ycombinator.com