Comment by simonw | Hacker News Reader

Comment by simonw | Hacker News Reader

Parent Full thread

The idea that LLMs are "running out of training data" is pretty widespread, but does it actually hold up?

Has anyone seen representatives of the cutting-edge AI labs - OpenAI, Anthropic, Meta, Mistral etc - express concern about this?

The impression I've been getting is that quality turns out to be much more important than quality. This tweet from Andrej Karpathy for example: https://twitter.com/karpathy/status/1797313173449764933

> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.

They don't express concerns because they've already gotten their data, and retroactive regulation won't ever take it away from them. You will never be able to train on even a fraction of the corpus OpenAI has culminated.

I suspect filtering out all the junk will make them smarter and smaller, of course. But then we will take all that saved space and pack it back full of other modalities like audio and images, and the cycle will continue for a long time yet.

Ideally a LLM would have every text in existence in its training set, right? I doubt we’re anywhere near that now.

Ideally a LLM would have every text in existence in its training set, right?

Only if every text in existence provides good source material to factor into generating the response. Given that half of all texts are even worse than average, this seems unlikely, and Karpathy’s argument seems very reasonable to me.

The final paragraph from the original tweet, after the quote in the GP comment, mentions another interesting aspect. Even if we do have sufficient expert-level source material in a particular field to train a useful generative AI model and that in itself produces more desirable responses than training on a larger but more average-quality data set, is there still potentially useful information that could be extracted from the larger training set as well? How can we classify which aspects of a larger training set are desirable to keep while filtering out noise that competes with higher-quality source material? It feels like the progress of generative AI over the next few years might be defined more by these kinds of questions than just trying to build ever larger models using ever larger sets of training data.

No, I don't think that would help. Andrej certainly doesn't seem to think so.

Reply on news.ycombinator.com