AI progress is driven by strong valuable data. Detailed important conversation with a chatbot are much more valuable that quick search queries. As LLMs extend past web UIs, there is even more interaction data to capture and learn from.
The company that captures the most human-AI interaction data will have a TREMENDOUS moat.
If I have the LLM translate a text from French to English... what is there to learn from that? Maybe the translation is great maybe it's awful, but there's no "correct" translation to evaluate the LLM against to improve it.
If I ask the chatbot for working code and it can't provide it, again, there's no "correct" code to train it against found in the conversation.
If I ask an LLM to interpret a bible passage, whether it does a good job or a terrible job there's no "correct" answer that the provider has to use as the gold standard, just the noise of people chatting with arbitrary answers.
When the big companies say they're running out of data, I think they mean it literally. They have hoovered up everything external and internal and are now facing the overwhelming mediocrity that synthetic data provides.
>> When the big companies say they're running out of data, I think they mean it literally. They have hoovered up everything external and internal and are now facing the overwhelming mediocrity that synthetic data provides.
> Digital data are only a tiny part of the influx of information that people interact with.
So what people bothers writing down are just a pale reflection of what has been, the reader has to relies on his experience and imagination to recreate it. If we take drawing for example, you may read all the books on the subject, you still have to practice to properly internalize that knowledge. Same with music, or even pure science (the axioms you start with are grounded in reality).
I believe LLMs are great at extracting patterns from written text and other forms of notation. They may be even good at translating between them. But as anyone who is polyglot may attest, literal translation is often inadequate because lot of terms are not equivalent. Without experiencing the full semantic meaning of both, you'll always be at risk at being confusing.
With traditional software, we were the ones providing meanings so that different tools can interact with each other (when I click this icon, a page will be printed out). LLMs are mostly translation machines, but with just a thin veneer of syntax rules and terms relationships, but with no actual meaning, because of all the information that they lack.
As another example, LLMs are kind of magical when it comes to what I'd call "bad memory spelunking". Is there a video game, book, or movie from your childhood, which you only have some vague fragments of, which you'd like to rediscover? Format those fragments into a request for a list of candidates, and if your description contains just enough detail, you will activate that semantic understanding to uncover what you were looking for.
I'd encourage you to check out 3blue1brown's LLM series for more on this!
I think it's true they lack a lot of information and understanding, and that they probably won't get better without more data, which we are running out of. That's sort of the point I was originally trying to make.
Is that actually true?