> They'll run out of data, and the models won't be able to get any better.
Well, that is enough to prove to me he doesn't know enough about the technology for me to pay attention to his opinions...
Well, that is enough to prove to me he doesn't know enough about the technology for me to pay attention to his opinions...
By this razor, I feel like I can dismiss your opinion. Can you elaborate why you think he doesn't know enough about technology for you to pay attention to his opinions? Is it cause you have proof that this is false? Cause it's predicting the future, there are an infinite number of ways it can go.
- internet (incl. blogosphere, all www, forums, social networks, dark net,...)
- all of digitized books
- all statistical data, governments data, military data
- all scientific papers and books
- all commercial companies data
- all private data/personal data
- ADDED - all video/audio data
- all new data...
If someone has some links on what the current state of affairs is (what they really trained on), I would be really interested.
I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means.
Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes).
Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
Post your counterargument. This is HN after all.
There is nothing that says you can't make much better models using the exact same data by changing the architecture of the network. In fact, that's exactly what we've been doing for the last 15 years.