- internet (incl. blogosphere, all www, forums, social networks, dark net,...)
- all of digitized books
- all statistical data, governments data, military data
- all scientific papers and books
- all commercial companies data
- all private data/personal data
- ADDED - all video/audio data
- all new data...
If someone has some links on what the current state of affairs is (what they really trained on), I would be really interested.
I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means.
Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes).
Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.