The Curse of Recursion: Training on generated data makes models forget (2023)

> The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

TL;DR: This paper confirms that Model Collapse can happen if the original data is replaced with synthetic data, but if both are used alongside each other, it no longer happens.

Catchy! And a really good point.

Seems like there could be room for a couple of special situations with caveats though? With the GAN formulation your generator can be practically as good as your discriminator and your discriminator can probably be better than it would have been without adversarial regularization?

isn't it obvious ? synthetic data was nice to generate an underlying distribution and avoid jumps in 'data holes', but with methods like diffusion now it's not as necessary

I'd argue that the case you give isn't an example of using a computer to generate data, it's a case of a human adding data (the data being the fact that the binary classifier should have a 50/50 balance).

This sort of massaging of data has its drawbacks as well--obviously this only works if the balance of that binary classifier actually is 50/50 in reality: I don't know enough about your case to say you were wrong, but I can imagine a lot of scenarios where a binary classifier should not be represented 50/50 in the data.

Just curious, but did you compute that 99% using purely real test data, or does your test set also include artificial data?

Can you clarify what you mean by using an LLM to “get a list that would even out the numbers”? If you’re doing binary classification, you need datapoints for features as well as the target class so how does an LLM synthetically create that without causing problems?

> Does it mean that data hungry corporation like Google, Facebook, Amazon, openai with Microsoft backing, that are already all around the internet and our phone tracking us have an incredibly advantage over open source model?

Yes, absolutely. Back around 2017 The Economist had an article calling "data is the new oil", I first heard that from a VC back in 2010.

These companies are sitting on immense reserves of data, Google, Facebook, Amazon, Bytedance are the Saudi Arabia, UAE, etc. of the information age.

Isn't everybody always gushing about how LLMs are supposed to get better all the time? If that's true then detecting generated fluff will be a moving target and an incessant arms race, just like SEO. There is no escape.

Right. Google already has a solution https://deepmind.google/technologies/synthid/ Everyone insists on training theirs to look human generated so the horses have left the stable on this

To me it seems intuitive that training on any unseen word patterns should increase intelligence as long as said patterns are consistent with ground truth. That's why it's counter-intuitive (to me) that training can fail, purely based on where the training data came from. The source of the information is something only the universe itself should be able to take into consideration (full causality chain), and not the training process.

> we currently do not have the ability to have an LLM learn from another LLM

We do. It's called model distillation, and it's relatively straightforward.

In fact, training a smaller model on the outputs of a much bigger model will significantly cut down on your training time/create a higher quality model than just training on raw human data (which is often low quality and noisy).

It has existed for years:

Self-Instruct: Aligning Language Models with Self-Generated Instructions https://arxiv.org/abs/2212.10560

airoboros: using large language models to fine-tune large language models https://github.com/jondurbin/airoboros

It's not a real problem, in my understanding. It's a "this kills cancer in a petri dish" sort of thing.

Yes, it makes sense that if your algorithm is at all lossy, passing outputs through it again compounds the loss.

The reality though that this doesn't happen on a large scale. JPEG isn't destroying all imagery because we're not stupid enough to constantly compound compression losses.

Most AI output is generated for some sort of purpose. If we're making images then we're throwing out the bad ones, retouching blemishes, and also still making new works entirely by hand.

Also the data processing inequality: https://en.wikipedia.org/wiki/Data_processing_inequality?use...

Relevant XKCD:

https://xkcd.com/1683/

That's assuming that the same function is applied in the same way at each iteration.

Think about this: The sum total of the human-generated knowledge was derived in a similar manner, with each generation learning from the one before and expanding the pool of knowledge incrementally.

Simply adding a bit of noise and then selecting good outputs after each iteration based on a high-level heuristic such as "utility" and "self consistency" may be sufficient to reproduce the growth of human knowledge in a purely mathematical AI system.

Something that hasn't been tried yet because it's too expensive (for now) is to let a bunch of different AI models act as agents updating a central wikipedia-style database.

These could start off with "simply" reading every single text book and primary source on Earth, updating and correcting the Wikipedia in every language. Then cross-translate from every source in some language to every other language.

Then use the collected facts to find errors in the primary sources, then re-check the Wikipedia based on this.

Train a new generation of AIs on the updated content and mutate them slightly to obtain some variations.

Iterate again.

Etc...

This could go on for quite a while before it would run out of steam. Longer than anybody has budget for, at least for now!

If you repeatedly apply one of three simple functions picked at random you might end up with Sierpinski triangle.

What about something like image improvement algorithms or NeRFs? They seem to increase information even if some of it is made up.

So correct me if I’m wrong here but wouldn’t another way to look at this be something like re-compressing a JPEG? Each time you compress a compressed jpeg you strip more and more information out of it? Same with any lossy compression, really.

These LLM’s are inherently a bit like lossy compression algorithms. They take information and pack it in a way that keeps its essence around (at least that is the plan). But like any lossy compression, you cannot reconstruct the original. Training a lossy compression scheme like an LLM using its own data is just taking that already packed information and degrading it.

I hope I’m right framing it this way because ultimately that is partly what an LLM is, it’s a lossy compression of “the entire internet”. A lossless model that can be queried like an LLM would be massive, slow and probably impossible with today’s tech.

I suspect that we will develop new information theory that mathematically proves these things can’t escape the box they were trained in, meaning they cannot come up with new information that isn’t already represented in the relationships between the various bits of data they were constructed with. They can “only” find new ways to link together the information in their corpus of knowledge. I use “only” in quotes because simply doing that alone is pretty powerful. It’s connecting the dots in ways that haven’t been done before.

Honestly the whole LLM space is cool as shit when you really think about it. It’s both incredibly overhyped yet very under hyped at the same time.

> with a non-reversible function f into f(x) then you are losing information.

A non-reversible function f does not necessarily lose information. Some non-reversible functions, like one-way functions used in cryptography, can be injective or even bijective but are computationally infeasible to invert, which makes them practically irreversible while retaining all information in a mathematical sense. However, there is a subset of non-reversible functions, such as non-injective functions, that lose information both mathematically and computationally. It’s important to distinguish these two cases to avoid conflating computational irreversibility with mathematical loss of information.

This seems obvious, but you're forgetting the inputs may actually have low entropy to begin with. Lossy compression is non-reversible, but usually the expectation is that we don't care about the parts we lost.

How might this cash out with recursive LLMs? Generalizing is very similar to compression: imagine recovering the Schrodinger equation from lots of noisy physical experiments. You might imagine that an LLM could output a set of somewhat general models from real data, and training it on data generated from those models generalizes further in future passes until maybe it caps out at the lowest entropy model (a theory of everything?)

It doesn't seem like it actually works that way with current models, but it isn't a foregone conclusion at the mathematical level at least.

It’s not intuitively obvious losing information makes things worse. In fact, it’s not even true. Plenty of lossy functions make the problem under consideration better off, such as denoising, optimizing, models that expose underlying useful structures, and on and on.

Also injecting noise can improve many problems, like adding jitter before ADC (think noise shaping, which has tremendous uses).

So claiming things like “can only make the end result worse” is “intuitive obvious” is demonstrably wrong.

> Additionally, humans create simulated stories like Dune, Lord of the Rings, or Harry Potter, which introduce fictional concepts, yet these stories still result in trainable data.

No, they don't, not in any sense where they are "simulated data". Dune is simulated data about what life would be like on Arrakis, and if you train a model to make predictions about that question, your model will be worthless trash. (Doesn't matter whether you train it on Dune or not.) Dune is real data about how English is used.

> humans create simulated stories like Dune, Lord of the Rings, or Harry Potter

People really anthropomorphize LLM to a full circle, don't they?

this is not a serious argument, please forgive me for saying

Thank you for making this comment, because it exposes some logical gaps.

Firstly, Go, Chess, and other games have objective rules and win criteria. (There is no “subjective opinion” as to whether Fischer or Spassky won their match.)

Language, the output of LLMs, does not have an objective function. Consider the following to sentences:

“My lips, two blushing pilgrims, ready stand.”

“My red lips are ready to kiss your red lips.”

Both are grammatically correct English sentences and both mean basically the same thing, but clearly the former by Shakespeare has a subjective poetic quality which the latter lacks. Even if we make evaluation rules to target (for example, “do not repeat phrases”, “use descriptive adjectives”, etc.) AI still seems to favor certain (for example “delve”) that are valid but not commonly used in human-originated English. There is then a positive feedback loop where these preferences are used to further train models, hence the next generation of models have no way of knowing whether the now frequent usage of “delve” is a human-originated or AI-originated phenomenon.

Lastly, regarding works of fiction, the concern is less about the content of stories—though that is also a concern—but more about the quality of language. (Consider above alternate take on Romeo and Juliet, for example.)