Hacker News Reader

Comment by mcswell | Hacker News Reader

The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.

View on HN

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be.

Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement.

OpenAI are licensing data from sources like the Associated Press.

Andrej Karpathy said this: https://twitter.com/karpathy/status/1797313173449764933

> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

I think this is roughly correct. My 2c is that folks used the initial web data to cold start and bootstrap the first few models, but so much of the performance increase we have seen at smaller sizes is a shift towards more conscientious data creation/purchase/curation/preparation and more refined evaluation datasets. I think the idea of scraping random text except maybe for the initial language understanding pre-training phase will be diminished over time.

This is understood in the academic literature as well, as people months/years ago were writing papers that a smaller amount of high quality data, is worth more than a large amount of low quality data (which tracks with what you can pick up from an ML 101 education/training).

You trim, yes, but AI content surely invades (all?) areas of written material. People are increasingly using AI to assist their writing. Even it if's for slight editing, word choice suggestions.

Even AP doesn't ban the use of LLMs, its standards prohibit direct publishing of AI-generated content. I'm sure its writers leverage LLMs in some ways in their workflow, though. They would probably continue to use these even if AP attempted to ban LLMs (human incentives).

If the AI generated content is filtered for quality or is corrected then it will still be good data. The phenomenon of model degradation is only in the case where there is no outside influence in the generated data.

I think this is extremely important with AI generated content, but seems to be given less and less thought as people start to "trust" AI as it seeps into the public conscious more. It needs to be reviewed, filtered, and fixed where appropriate. After that, it isn't any different from reviewing data on your own, and wording it in a way that fits the piece you're writing. Unfortunately, there's so much trust in AI now that people will go ahead and publish content without even reading it for the correct tense!

The same problem exists if you blindly trust any source without verifying it. There is a huge amount of endlessly recycled incorrect blog spam out there for all domains. Not only that but this problem has always existed for second hand information so it's not like we were even starting from some pristine state of perfect truthfulness. We have the tools we need to deal with the situation and they were developed hundreds of years ago. Empiricism being chief among them. Nullius in verba[0]

[0] https://en.wikipedia.org/wiki/Nullius_in_verba

If tail events aren't produced by these models, no amount of human filtering will get them back. People would not just need to filter or adjust AI generated content, but create novel content of their own.

> The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

Perhaps we should stop exposing humans to them, as well?

Absolutely for learning. If you want to learn something we should realize it's awful. Try to learn something in your own field to see how awful it is.

That's why we're all armchair experts in other domains.

They are moving to Discord.

It’s already the case that people don’t see that stuff very much.

The key word in that quote is “average.” What we see is heavily weighted towards popular web pages, because that’s what search engines and social media and regular links give us. We don’t see average.

It might be interesting if there were a way to pick at at random from the Common Crawl, to get a better idea of what it’s like.

This is a similar problem to what was observed in Diffusion models going "MAD" when trained on synthetic data. https://arxiv.org/abs/2307.01850 . Therefore, going forward AI companies will find it increasingly difficult to get their data by scraping the. web, because web will be full of synthetically generated data.

They needed to deal with degenerate data on the Web anyway. It's always been full of trash and spam.

I agree with you when it comes to training, but at the same time, I think that's also the power we get with the web. You can have a voice, even if others don't agree with you. I don't think that should be taken away unless you are inciting violence.

At least some of the LLM generated content will be vetted/selected for by a human being though.

Read the paper, the problem is each generation forgets information. Starting at the tails of the distribution they learn. No amount of filtering/selecting would help here. People would need to fill in missing information without AI help. If they are just filtering, it does nothing to stop model collapse.

The experiment in the paper is not well designed. They are repeatedly fine tuning the model and replacing the entire data set each time with a noisier version. That's just not how the world works and is literally the most naive approach you could take. They should have attempted to increase the size of the training set using output from the model combined with human editing and input and figured out a good evaluation strategy. That would have at least approached reality and may have produced useful knowledge. The fact still remains that the paper is hopelessly far behind the sota and almost entirely divorced from the processes it intends to make claims about.

Reply on news.ycombinator.com