Are 99% of LLMs based on web scraping?
I just get this impression when reading news on recent lawsuits involving Google bot "fair use" of IP protected scraped data, OpenAI, and Midjourney
How and where do you get most of human knowledge nowadays ?
As I estimate, there are few terabytes of books in library of Congress, and this considered largest books collection in whole world.
Fortunately, many books in LoC already public domain (copyright expired), so could use without limitations. For others, things are complicated.
In overall, scientists allowed to use any book at their work, but this mean automatically, that derived content could not be used commercially, only for research or for arts.
On other side, people tried to use synthetic datasets, generated by algorithms, or even got real texts and just dilute with noise words. But unfortunately this leads to hallucinations.
So, if you need more data then in PD books to train model, you have to scrape web, or to make some tricks, like use PD videos (that's why GPT-4 become multimodal), or hire lot of people and pay them to talk with model (this is just very expensive).
I hear, some big corporations feed models with their own internal talks, for example used records of support talking with customers, but this is too synthetic approach and very limited.