That's more or less what people were doing back in 2023 - crawling everything and dumping as much data in as possible.
It's not a great strategy to build a best-in-class model though, as a lot of the internet is junk. The SolidGoldMagikarp/davidjl bug is the kind of thing that happens if you crawl all of https://www.reddit.com/r/counting/ for example: https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-...
These days model training labs are more selective about what they train on. Most of the game of training a great model comes down to selectively training your data. They still use a lot of unlicensed data but it's a bit more sophisticated than just dumping in everything they can find.
There was a recent paper [1] that showed that for one particular model it was possible to get very good reasoning performance by finetuning on under a thousand examples of reasoning.
and models are using synthetic data sets now, for like 2 years
as in, the huge models did the crawling and then the next models had conversations with the huge models
oversimplified but basically the scraping ship has sailed, artists/writers will never be satisfied, and other companies bragging about using whatever aspirational goal post of licensed material are just doing it for brownie points
eg if I ask DeepSeek to write a story it tends to write the bullet points outline of a story. And some other models always give characters the same names.
for now, they're not doing that but just releasing new models that also have updated information
there is one group thats scraping the latest information and including it in their latest models, another group thats only doing synthetic data sets from the prior group's models, and every enthusiast everywhere making fine tuned LLMs with synthetic data sets and whatever niche they want
R1 didn’t train a base model, they performed additional steps on top of a previously-trained base model (V3). These guys are doing something similar.