I don't get this argument. The paper is about "whether RLLMs can think". If we grant "humans make these mistakes too", but also "we still require this ability in our definition of thinking", aren't we saying "thinking in humans is a illusion" too?
"By AGI, we mean highly autonomous systems that outperform humans at most economically valuable work."
AWS: https://aws.amazon.com/what-is/artificial-general-intelligen...
"Artificial general intelligence (AGI) is a field of theoretical AI research that attempts to create software with human-like intelligence and the ability to self-teach. The aim is for the software to be able to perform tasks that it is not necessarily trained or developed for."
DeepMind: https://arxiv.org/abs/2311.02462
"Artificial General Intelligence (AGI) is an important and sometimes controversial concept in computing research, used to describe an AI system that is at least as capable as a human at most tasks. [...] We argue that any definition of AGI should meet the following six criteria: We emphasize the importance of metacognition, and suggest that an AGI benchmark should include metacognitive tasks such as (1) the ability to learn new skills, (2) the ability to know when to ask for help, and (3) social metacognitive abilities such as those relating to theory of mind. The ability to learn new skills (Chollet, 2019) is essential to generality, since it is infeasible for a system to be optimized for all possible use cases a priori [...]"
The key difference appears to be around self-teaching and meta-cognition. The OpenAI one shortcuts that by focusing on "outperform humans at most economically valuable work", but others make that ability to self-improve key to their definitions.
Note that you said "AI that will perform on the level of average human in every task" - which disagrees very slightly with the OpenAI one (they went with "outperform humans at most economically valuable work"). If you read more of the DeepMind paper it mentions "this definition notably focuses on non-physical tasks", so their version of AGI does not incorporate full robotics.
General-Purpose (Wide Scope): It can do many types of things.
Generally as Capable as a Human (Performance Level): It can do what we do.
Possessing General Intelligence (Cognitive Mechanism): It thinks and learns the way a general intelligence does.
So, for researchers, general intelligence is characterized by: applying knowledge from one domain to solve problems in another, adapting to novel situations without being explicitly programmed for them, and: having a broad base of understanding that can be applied across many different areas.
If something can be better than random chance in any arbitrary problem domain it was not trained on, that is AGI.
Since there's not really a whole lot of unique examples of general intelligence out there, humans become a pretty straightforward way to compare.
No so unconventional in many cultures.
In this case, I was thinking of unusual beliefs like aliens creating humans or humans appearing abruptly from an external source such as through panspermia.
If somebody claims "computers can't do X, hence they can't think". A valid counter argument is "humans can't do X either, but they can think."
It's not important for the rebuttal that we used humans. Just that there exists entities that don't have property X, but are able to think. This shows X is not required for our definition of "thinking".
Or perhaps AGI should be able to reach the level of an experienced professional in any task. Maybe a single system can't be good at everything, if there are inherent trade-offs in learning to perform different tasks well.
It's surprisingly simple to be above average in most tasks. Which people often confuse with having expertise. It's probably pretty easy to get into the 80th percentile of most subjects. That won't make you the 80th percentile of people that do the thing, but most people don't. I'd wager 80th percentile is still amateur.
But only the limited number of tasks per human.
> Or perhaps AGI should be able to reach the level of an experienced professional in any task.
Even if it performs just better than untrained human but on any task this will be superhuman level. As no human can do it.
Models still have extreme limits relative to humans. Context size and reasoning depths, being the two most obvious. A third being their inability to incorporate new information with as little effort as humans do, without creating unintended conflicts across previously learned information.
But they vastly exceed human capabilities in other ways. The most obvious, being their ability to do shallow reasoning incorporating information from virtually any combination out of the vast number of topics that humans find useful or interesting. Another being their ability to by default produce discourse with such high written organization and grammatical quality.
For now, they are artificial "better at different things" intelligences.
But yes, you’re right that software needs not be AGI to be useful. Artificial narrow intelligence or weak AI (https://en.wikipedia.org/wiki/Weak_artificial_intelligence) can be extremely useful, even something as narrow as a services that transcribes speech and can’t do anything else.
The implication here is that they excel at things that occur very often and are bad at novelty. This is good for individuals (by using RLMs I can quickly learn about many other aspects of human body of knowledge in a way impossible/inefficient with traditional methods) but they are bad at innovation. Which, honestly, is not necessarily bad: we can offload lower-level tasks[0] to RLMs and pursue innovation as humans.
[0] Usual caveats apply: with time, the population of people actually good at these low-level tasks will diminish, just as we have very few Assembler programmers for Intel/AMD processors.
Find me one that can solve it entirely in their head without touching the actual thing and externalizing state.
What happens when some novel Tower of Hanoi-esque puzzle is presented and there's nothing available in its training set to reference as an executable solution? A human can reason about and present a solution, but an LLM? Ehh...
Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties" is something "not in" the training set, but it's really just a composition of the "make a linked list class" and "sort these things based on a field" problems.
Yes, knowledge is compositional. This is just as true for humans as it is for machines.
We reason about things based on our training data. We have a hard time or impossible time reasoning about things we haven’t trained on.
Ie: a human with no experience of board games cannot reason about chess moves. A human with no math knowledge cannot reason about math problems.
How would expect an LLM to reason about something with no training data?
Then how did the first humans solve math and chess problems, if there were none around solved to give them examples of how to solve them in the first place?
Also the idea of "problems" like "chess problems" and "math problems" is itself constructed. Chess wasn't created by stacking together enough "chess problems" until they turned into a game - it was invented and tuned as a game for a long time before someone thought about distilling "problems" from it, in order to aid learning the game; from there, it also spilled out into space of logical puzzles in general.
This is true of every skill, too. You first have people who master something by experience, and then you have others who try to distill elements of that skill into "problems" or "exercise regimes" or such, in order to help others reach mastery quicker. "Problems" never come first.
Also: most "problems" are constructed around a known solution. So another answer to "how did the first humans solve" them is simply, one human back-constructed a problem around a solution, and then gave it to a friend to solve. The problem couldn't be too hard either, as it's no fun to not be able to solve it, or to require too much hints. Hence, tiny increments.
Based upon that comprehension, we then need little working memory (tokens) to solve the problem, it just becomes tedious to execute the algorithm.. But the algorithm was derived after considering the first 3 or 4 cases.
Whereas for the moment, LLMS are just pattern matching; whereas we do the pattern match, then derive the generalised rule.
The Tower of Hanoi problem is terrible example for somehow suggesting humans are superior.
Firstly, there are plenty of humans who can’t solve this problem even for 3 disks, let alone 6 or 7. Secondly, LLMs can both give you general instructions to solve for any case and they can write out exhaustive move lists too.
Anyway, the fact that there are humans who cannot do Tower of Hanoi already rules it out as a good test of general intelligence anyway. We don’t say that a human doesn’t have “general intelligence” if they cannot solve Towers of Hanoi, so why then would it be a good test for LLM general intelligence?
This has not been my experience. They might do something in the right direction. They might write complete garbage. But the amount of time an LLM writes code that compiles and executes first time is vanishingly few for me. Perhaps I'd have better luck if I were doing things which weren't _actual_ niche problems.
This already excludes a lot of humans
The paper doesn't give any evidence humans are able to do this. And I honestly find it very implausible. Even Gary Marcus admits in (1) that humans would probably make mistakes.
I think the answer to this question is certainly "Yes". I think the reason people deny this is because it was just laughably easy in retrospect.
In mid-2022 people were like. "Wow this GPT3 thing generates kind of coherent greentexts"
Since then really only we got: larger models, larger models, search, agents, larger models, chain-of-thought and larger models.
And from a novelty toy we got a set of tools that at the very least massively increase human productivity in a wide range of tasks and certainly pass any Turing test.
Attention really was all you needed.
But of course, if you ask a buddhist monk, he'll tell you we are attention machines, not computation machines.
He'll also tell you, should you listen, that we have a monkey in our mind that is constantly producing new thoughts. This monkey is not who we are, it's an organ. It's thoughts are not our thoughts. It's something we perceive. And that we shouldn't identify with.
Now we have thought-genrating-monkeys with jet engines and adrenaline shots.
This can be good. Thought-genrating-monkeys put us on the moon and wrote Hamlet and the Oddesy.
The key is to not become a slave to them. To realize that our worth consists not in our ability to think. And that we are more than that.
I cannot afford to consider whether you are right because I am a slave to capital, and therefore may as well be a slave to capital's LLMs. The same goes for you.
I get too hot in summer and too cold in winter. I die of hunger. I am harassed by critters of all sorts.
And when my bed breaks, to keep my fragile spine from straining at night, I _want_ some trees to be cut, some mattresses to be provisioned, some designers to be provisioned etc. And capital is what gets me that, from people I will never meet, who wouldn't blink once if I died tomorrow.
But the first civilizations in the world around 3000BC had trade, money, banking, capital accumulation, divison of labour etc.
In small tribes, where everyone knew everyone intimately because they lived together, and everything was managed by feels.
Things like rules, laws, money, banking, hierarchies, well-defined private vs. public ownership, are all things that came with scale, because interpersonal relationships fail to keep group cohesion once it reaches more than ~100 people.
It is unequivocally "No". A good joint distribution estimator is always by definition a posteriori and completely incapable of synthetic a priori thought.
Now let's say you didn't know the true function and had to use a neural network instead. You would probably still get a great result in the sense of generating "new" outputs that are not observed in the training data, as long as they are within or reasonably close to the original domain.
LLMs are that. With enough data and enough parameters and the right inductive bias and the right RLHF procedure etc, they are getting increasingly good at estimating a conditional next token distribution given the context. If by "synthetic" you mean that an LLM can never generate a truly new idea that was not in it's training data, then that becomes the question of what the "domain" of the data really is.
I'm not convinced that LLMs are strictly limited to ideas that they have "learned" in their data. Before LLMs, I don't think people realized just how much pattern and structure there was in human thought, and how exposed it was through text. Given the advances of the last couple of years, I'm starting to come around to the idea that text contains enough instances of reasoning and thinking that these models might develop some kind of ability to do something like reasoning and thinking simply because they would have to in order to continue decreasing validation loss.
I want to be clear that I am not at all an AI maximalist, and the fact that these things are built largely on copyright infringement continues to disgust me, as do the growing economic and environmental externalities and other problems surrounding their use and abuse. But I don't think it does any good to pretend these things are dumber than they are, or to assume that the next AI winter is right around the corner.
You don't seem to understand what synthetic a priori means. The fact that you're asking a model to generate outputs based on inputs means it's by definition a posteriori.
>You would probably still get a great result in the sense of generating "new" outputs that are not observed in the training data, as long as they are within or reasonably close to the original domain.
That's not cognition and has no epistemological grounds. You're making the assumption that better prediction of semiotic structure (of language, images, etc.) results in better ability to produce knowledge. You can't model knowledge with language alone, the logical positivists found that out to their disappointment a century or so ago.
For example, I don't think you adequately proved this statement to be true:
>they would have to in order to continue decreasing validation loss
This works if and only if the structure of knowledge lies latently beneath the structure of semiotics. In other words, if you can start identifying the "shape" of the distribution of language, you can perturb it slightly to get a new question and expect to get a new correct answer.
The fact that the human mind can think in concepts, images AND words, and then compresses that into words for transmission, wheras LLMs think directly in words, is no object.
If you watch someone reach a ledge, your mind will generate, based on past experience, a probabilistic image of that person falling. Then it will tie that to the concept of problem (self-attention) and start generating solutions, such as warning them or pulling them back etc.
LLMs can do all this too, but only in words.
Quick aside here: They do not think. They estimate generative probability distributions over the token space. If there's one thing I do agree with Dijkstra on, it's that it's important not to anthropomorphize mathematical or computing concepts.
As far as the rest of your comment, I generally agree. It sort of fits a Kantian view of epistemology, in which we have sensibility giving way to semiotics (we'll say words and images for simplicity) and we have concepts that we understand by a process of reasoning about a manifold of things we have sensed.
That's not probabilistic though. If we see someone reach a ledge and take a step over it, then we are making a synthetic a priori assumption that they will fall. It's synthetic because there's nothing about a ledge that means the person must fall. It's possible that there's another ledge right under we can't see. Or that they're in zero gravity (in a scifi movie maybe). Etc. It's a priori because we're making this statement not based on what already happened but rather what we know will happen.
We accomplish this by forming concepts such as "ledge", "step", "person", "gravity", etc., as we experience them until they exist in our mind as purely rational concepts we can use to reason about new experiences. We might end up being wrong, we might be right, we might be right despite having made the wrong claims (maybe we knew he'd fall because of gravity, however there was no gravity but he ended up being pushed by someone and "falling" because of it, this is called a "Gettier problem"). But our correctness is not a matter of probability but rather one of how much of the situation we understand and how well we reason about it.
Either way, there is nothing to suggest that we are working from a probability model. If that were the case, you wind up in what's called philosophical skepticism [1], in which, if all we are are estimation machines based on our observances, how can we justify any statement? If every statement must have been trained by a corresponding observation, then how do we probabilistically model things like causality that we would turn to to justify claims?
Kant's not the only person to address this skepticism, but he's probably the most notable to do so, and so I would challenge you to justify whether the "thinking" done by LLMs has any analogue to the "thinking" done using the process described in my second paragraph.
[1] https://en.wikipedia.org/wiki/Philosophical_skepticism#David...
When I spill a drink, I don't think "gravity". That's too slow.
And I don't think humans are particularly good at that kind of rational thinking.
I think you do, you just don't need to notice it. If you spilled it in the International Space Station, you'd probably respond differently even if you didn't have to stop and contemplate the physics of the situation.
So we receive inputs from the environment and cluster them into observations about concepts, and form a collection of truth statements about them. Some of them may be wrong, or apply conditionally. These are probabilistic beliefs learned a posteriori from our experiences. Then we can do some a priori thinking about them with our eyes and ears closed with minimal further input from the environment. We may generate some new truth statements that we have not thought about before (e. g. "stepping over the ledge might not cause us to fall because gravity might stop at the ledge") and assign subjective probabilities to them.
This makes the a priori seem to always depend on previous a posterioris, and simply mark the cutoff from when you stop taking environmental input into account for your reasoning within a "thinking session". Actually, you might even change your mind mid-reasoning process based on the outcome of a thought experiment you perform which you use to update your internal facts collection. This would give the a priori reasing you're currently doing an even stronger a posteriori character. To me, these observations above basically dissolve the concept of a priori thinking.
And this makes it seem like we are very much working from probabilistic models, all the time. To answer how we can know anything: If a statement's subjective probability becomes high enough, we qualify it as a fact (and may be wrong about it sometimes). But this allows us to justify other statements (validly, in ~ 1-sometimes of cases). Hopefully our world model map converges towards a useful part of the territory!
I think not, we can get close, but there exists problems and situations beyond that, especially in mathematics and philosophy. And I don't a visual medium or combination of is sufficient either, there's a more fundamental, underlying abstract structure that we use to model reality.
It's sufficient to the level needed for human intelligence. We're a product of evolution, and we only need as much abstraction as it's required for operational reasons. Modeling reality in a deep, abstract way is something we want to, but not something that was required for our minds to evolve, nor for us to create civilization as it is today.
After much time trying to accomplish this during the 20th century, the answer was as resounding "no" [1]
[1] https://en.wikipedia.org/wiki/Logical_positivism#Decline_and...
The argument is that LLMs are computer systems and a computer system that's as bad as a human is less useful than a human.