LLMs are plausibility engines. The fundamental hypothesis on trial here is that increasing plausibility corresponds to increasing correctness. This is easily rejected for the human-sourced content that trains them, implying an upper bound on any dependent phenomenon. It follows that simply scaling LLMs will not produce AGI.
This is one way to look at LLMs. But it doesn't automatically impose an upper bound on their capabilities.
Humans are reproductive organisms. This is also true. At a glance, it would seem that humans could not evolve to be intelligent. They are only selected on their reproductive abilities. But obviously the categorization, while true, doesn't impose an upper bound on human abilities.
LLMs are evolved to be as efficient as possible at retaining knowledge.
A simple strategy for retaining knowledge is memorization. Neural networks definitely can do that.
A different strategy for retaining knowledge is using algorithms. Neural networks can evolve to retain knowledge using algorithms too. For instance, it has been shown that a small neural network evolved FFT-like structures to perform addition. It started out with memorization, which was far from perfect, but later in the training it switched to an algorithm using FFT for addition which produced perfect results.
I think the better LLMs retain knowledge through sophisticated compression. It involves building a world model and ways to relate the input text to that model.
I see it as the building blocks of a reasoning machine. It is incomplete and buggy, and the current architectures might hit a ceiling soon, but nonetheless it is a completely different thing than pure memorization.
Note that OpenAI admits that pre-training (pattern finding and matching) is where the abilities on professional tests comes from.
Even proofs in presburger arithmetic have an upper bound on the second level of the polynomial hierarchy, which is way beyond the computational power of any LLM
https://arxiv.org/abs/2207.00729
>The model’s capabilities on exams appear to stem primarily from the pre-training process and are not significantly affected by RLHF. On multiple choice questions, both the base GPT-4 model and the RLHF model perform equally well on average across the exams we tested
https://arxiv.org/abs/2303.08774
To practically scale, parallelism is required, and the ability to find algorithms within the L or TC0 complexity classes is limited.
Remember that individual ANNs as used in transformers are just binary linear classifiers. Which you can do a lot with, but probably limited to P assuming we don't find out L or TC0 are larger than we think now.
Presburger arithmetic, or first order logic with (+,=), or (*,=) is the strongest FoL that is both complete and consistent. move up to Peano arithmetic and you start to hit the limits from Gödel, Church, and Turing.
There will be instances that an LLM can learn with CoT, zero shot etc...
But most of those will be due to luck or parallel patterns in the training set.
LLMs simplicity bias is great for plausible answers to our of distribution questions, but puts limits on what can be learned as far as 'algorithms' go.
They will work for 'there exists' problems far better than for any 'for all' or 'for most' problems unless they have learnable patterns in the training set.
The whole scientific system specifically being designed to suppress plausible-sounding-but-incorrect claims says otherwise.
But that we have came up with a scientific system is proof that humans actually can tell the difference, we might be bad at it but we still manage to do it. Humans that can't tell the difference are very unreliable and usually shunned in engineering and other STEM fields.
Furthermore, sometimes mistakes are made when assessing correctness, even in mathematical proofs. A correctness judgement thus still has a degree of plausibility associated with it.
I'm thus not sure if the distinction being made here isn't just one of degree.
I think humans has two levels here, "that sounds plausible" and "I can't imagine how this could be wrong so I am willing to risk my life on it". The second can still be wrong, but usually it isn't. For example when you walk on a bridge, you assume the bridge is there and confidently put down your foot even though you would die if that part of the bridge happened to be an hallucination.
So science is a set of statements and experiments that sums up to facts that humans can't imagine are wrong. Sometimes they are still a bit wrong, but then only slightly like newtons equations or theories that aren't backed up by experiments. But in general science is full of facts that humans stake their life on every day.
LLMs doesn't seem to have that certainty layer, something so certain that the LLM would stake its life on it. I'd for sure never stake my life on anything an LLM says as it is today, but I happily do it for humans every time I take the elevator etc.
The scientific method is just a way to make humans use that reliable part of their brain instead of the plausible part, even in situations that aren't life or death. I don't think it is possible to come up with something like the scientific method without having a built-in sense of what facts are.
There is no such thing in 'higher intelligence' beyond protoception. Pretty much every mammal has the same built in sense that you do, and yet none of them are carting around human intelligence. More so, when human children are left feral and not raised around other humans we typically consider them mentally disabled. Again, they learn the protoception stuff just fine, but all the stuff that "puts us above" the mammals is missing, it is not inane. Your counting, reading, and tools use are all learned behaviors after birth and form a continuous chain of learned behaviors taught by people (and more recently books and computers).
There is a reason that after development of the scientific method that most human endeavors experienced an exponential growth curve. It's because we don't recognise facts from fiction at a much higher rate than other animals, it's that we teach each other fact from fiction better than other animals.
Having a sense of what are facts and what are assumptions doesn't mean you are smart, but not having it means you are pretty dumb. LLMs doesn't have that.
Natural language is more powerful than programming languages since it lets us communicate assumptions, guesses, uncertainty etc, to other intelligent beings. So a good LLM needs to be able to consider certainty level of statements and act accordingly, todays LLMs doesn't do that so generate dumber responses than a human would since they lack a sanity checker.
AI is on a path to outperform humans at a large number of tasks and jobs that previously would have been described as requiring intelligence.
the implication sounds good, but is easily rejected with a counter example: a good student can surpass his master. A mediorcre student with multiple masters can surpass all of them (tangent: this is why the medival master-journeyman system was so efficient, I think).
Or to make the argument more abstract: Your implication seems to assume, that transferlearning does not exist.
I’ll repeat an assertion I’ve made in this forum before: that the benchmark for AGI will be when two AIs play one another at an incomprehensible game of their own devising.
However, there is a hidden corollary in my earlier statement. By implying an upper bound on the correctness of their answers, we can assess that LLMs will not themselves achieve sentience and form a civilisational threat. That actually increases their potential utility in many applications. Much as you wouldn’t, for safety’s sake, put a jet engine in a conventional car (occasional mad-scientist projects notwithstanding).