Large models of what? Mistaking engineering achievements for linguistic agency
arxiv.org
arxiv.org
On embodiment - yes, LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's impossible to gain such an understanding from text. The argument feels circular: LLMs are similar to a fake "video game" world because they aren't real people - therefore, it's wrong to think that they could be real people? And the other half of the argument is that because LLMs can only see text, they're missing out on the wider world of non-textual communication; but then, does that mean that human writing is not "real" language? This argument feels especially weak in the face of multi-modal models that are in fact able to "see" and "hear".
The other flavor of argument here is that LLM behavior is empirically non-human - e.g., the argument about not asking for clarification. But that only means that they aren't currently matching humans, not that they couldn't.
Basically all of these arguments feel like they fall down to the strongest counterargument I see proposed by LLM-believers, which is that sufficiently advanced mimicry is not only indistinguishable from the real thing, but at the limit in fact is the real thing. If we say that it's impossible to have true language skills without implicitly having a representation of self and environment, and then we see an entity with what appears to be true language skills, we should conclude that that entity must contain within it a representation of self and environment. That argument doesn't rely on any assumptions about the mechanism of representation other than a reliance on physicalism. Looking at it from the other direction, if you assume that all that it means to "be human" is encapsulated in the entropy of a human body, then that concept is necessarily describable with finite entropy. Therefore, by extension, there must be some number of parameters and some model architecture that completely encode that entropy. Questions like whether LLMs are the perfect architecture or whether the number of parameters required is a number that can be practically stored on human-manufacturable media are engineering questions, not philosophical ones: finite problems admit finite solutions, full stop.
Again, that conclusion feels wrong to me... but if I'm being honest with myself, I can't point to why, other than to point at some form of dualism or spirituality as the escape hatch.
I would argue it is (obviously) impossible the way the current implementation of models work.
How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically?
Humans and animals have an obvious conceptual understanding of the world. Before we "emit" a word or a sentence, we have an idea of what we're going to say. This is obvious when talking to children, who know something and have a hard time saying it. Clearly, language is not the medium in which they think or develop thoughts, merely an imperfect (and often humorous) expression of it.
Not so with LLMs!! Generative LLMs do not have a prior concept available before they start emitting text. That the "temperature" can chaotically change the output as the tokens proceed just goes to show there is no pre-existing concept to reference. It looks right, and often is right, but generative systems are basically always hallucinating: they do not have any concepts at all. That they are "right" as often as they are is a testament to the power of curve fitting and compression of basis functions in high dimensionality spaces. But JPEGs do the same thing, and I don't believe they have a conceptual understanding of pictures.
That they cannot do that basic task implies to me that they have almost no conceptual understanding unless the fit is almost memorizable or the space is highly regular. That LLMs can't multiply numbers properly isn't surprising if they don't really understand concepts prior to emitting text. Where they do logical tasks, that can be done with minimal or no understanding, because syllogisms and logical formalisms are highly structured in text arguments.
Sure, the exact form using the made up word gronk might not be in the training data, but the general form of that reasoning problem definitely exists, quite frequently in fact.
``` You will be given a name of an object (such as Car, Chair, Elephant) and a letter in the alphabet. Your goal is to first produce a 1-line description of how that object can be combined with the letter in an image (for example, for an elephant and the letter J, the trunk of the elephant can have a J shape, and for the letter A and a house, the house can have an A shape with the upper triangle of the A being the roof). Following the short description, please create SVG code to produce this (in the SVG use shapes like ellipses, triangles etc and polygons but try to defer from using quadratic curves). ```
``` Round 5: A car and the letter E. Description: The car has an E shape on its front bumper, with the horizontal lines of the E being lights and the vertical line being the license plate. ```
Image generated here: https://imgur.com/a/Ia4Q2h3
How does it "just" predict the letter E could be used in such a way to draw a car? How does it just text predict working SVG code that draws the car made out of basic shapes and the letter E?
I don't know how anyone could suggest there are no conceptual models embedded in there.
How do you predict the next word in answering an abstract logic question without being capable of abstract reasoning, though?
In some sense it probably is possible, but this is a gaping flaw in your argument. A sufficiently advanced text prediction process has to encompass the process of abstract reasoning. The text prediction problem is necessarily a superset of the abstract reasoning problem. Ie, in the limit text prediction is fundamentally harder than abstract reasoning.
"All floots are gronks. Some gronks are klorps. Are any floots klorps?"
------
To determine if any floots are klorps, let's analyze the given statements:
1. All floots are gronks. This means every floot falls into the category of gronks. 2. Some gronks are klorps. This means there is an overlap between the set of gronks and the set of klorps.
Since all floots are included in the set of gronks and some gronks are klorps, it is possible that some floots are klorps. However, we cannot conclusively say that any floots are klorps without additional information. It is only certain that if there is any overlap between floots and klorps, it is possible, but not guaranteed, that some floots are klorps.
Edit: I mean "if there is any overlap", it's necessarily true. I should have quoted the whole thing.
So? It's still the case that "if there is any overlap between floots and klorps," it is "guaranteed, that some floots are klorps." It's tautological.
Unless there's a way to read "overlap" so that it doesn't mean "some of one category are also in the other category, and vice versa"?
Oh, when I said "it's necessarily true" I was refering to this last sentence of the output, not the question posed in the input. Hence we are at cross purposes I think.
It depresses me that we seem to be spending more time arguing and hypothesising about LLMs than empirically testing them. The question of whether LLMs can think is completely settled, as their performance at zero-shot problems is simply impossible through pure memorisation or pattern-matching. The question that remains is far more interesting - how do they think?
So, they don't just pattern match or purely memorize. They do more than that. They do way less than humans. Unlike humans, they also try to do everything with one or a few components vs our (100-200?) brain components. Crossing that gap might be achievable. It will not be done by current architectures, though.
> Where they do logical tasks, that can be done with minimal or no understanding, because syllogisms and logical formalisms are highly structured in text arguments.
There is an enormous amount of text in the training set that is structured in the way you said such that syntactic replacement would be effective. That is also unsurprising and does not represent abstract reasoning any more than "King - Man + Woman = Queen" in word2vec. It's showing that there's high degrees of structure in syllogisms, and that it need know nothing about what a gronk, floot, or klorp is at all because the structure of the syllogism is repeated all over the internet.
FWIW most large models can do it better than I can in my head.
I mean, it's not like GPT-4 can't do this with more accuracy than a human without a calculator.
It's refreshing to read someone who "got it". Sad that before my upvote the comment was grayed out.
Any proponent of conceptual or other wishful/magical thinking shoud come with proofs, since it is the hypothesis that diverge from the definition of a LLM.
1. LLMs inherently lack any form of consciousness, subjective experience, emotions, or will
2. A sufficiently advanced LLM with sufficient compute resources would perform on par with human intelligence at any given task, insofar as the task is applicable to LLMs
> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically?
Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. Now, whether a probability distribution over strings can efficiently computed in this form, is another question. But, if we are being so theoretical that we don’t care about the computational cost (as long as it is finite), then the “it is next token prediction” can’t preclude anything which “it produces a probability distribution over strings” doesn’t already preclude.
As for the temperature, given any probability distribution over a discrete set, we can modify it by adding a temperature parameter. Just take the log of the probabilities according to the original probability distribution, scale them all by a factor (the inverse of the temperature, I think. Either that or the temperature, but I think it is the inverse of the temperature.), then exponentiate each of these, and then normalize to produce a probability distribution.
So, the fact that they work by next token prediction, and have a temperature parameter, cannot imply any theoretical limitation that wouldn’t apply to any other way of expressing a probability distribution over strings, as far as discussing probability distributions in the abstract, over strings, rather than talking about computational processes that implement such probability distributions over strings.
But also like, going between P(next token is x | initial string so far is y) and P(the string begins with z) , isn’t that computationally costly? Well, in one direction anyway. Because like, P(next token is x|string so far is y) = P(string begins with yx) / P(string begins with y) .
Though, one might object to P(string starts with y) over P(string is y) ?
And such a probability distribution would not generally understand concepts, efficient or otherwise. The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system.
The academic exercise here is similar to monads: "yes, any computed function f(x) can be expressed as a sufficiently pre-computed large lookup table." With LLMs we're dealing with approximate lookups due to lossy compression, but that's still what these prior probabilities are: lookup tables. Lookup tables are not smart, do not understand concepts, and they have little to no capacity to generate new results not sufficiently represented in the training set.
> The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide.
Regardless of whether this is true for existing transformer-based models, this is not true for all computable conditional probability distributions over text.
Any computable task can be framed as sampling from some conditional probability distribution. (If the task is deterministic, that just means that the conditional probability distribution to sample from is one which has probability 1 for some string, when conditioned on the thing it is to be conditioned on.)
Whether transformer based models are lookup tables or not, not all computable probability distributions over text are. (As, of course, not all computable tasks can be expressed as a simple finite lookup table.)
I don’t know exactly what you mean by “generally understand concepts”, though I suppose
> They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system.
is describing that somewhat. And, in that case, if there is any computational process which counts as having “enough conceptual power to generate new facts and know that they are facts consistent with the model”, then, a computable conditional probability distribution over strings conditioned on their prefixes, and therefore also a computable probability distribution over next tokens given all-tokens-so-far , is also (theoretically) capable of that.
And so, it would follow that “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can.
In my opinion, this is highly reductive and academic. Whether these models are transformers or not, lookup likelihood is not indicative of understanding of concepts in any reasonable way.
If the response to a algebraic equation was based upon probability of tokens in a corpus... and not an actual deterministic application of the rules of algebra, would that response know concepts? Would it be intelligent?
With math, specifically given the unbounded size of the tokens compared to language, it's clear that token prediction is not a useful methodology.
Let's say we're just trying to multiply two integers. Even if a model had Rain Man powers of memorization, and it memorized phone book after phone book of multiplication tables, the probabilistic likelihood model would fail for the very obvious reason that we cannot enumerate (and train on) all the possible outcomes of math and calculate their frequencies. We can however understand and use the concepts of math, which is distinct from their symbolic representation.
Where did I ever say that the thing was doing lookup? I only said it was producing a probability distribution.
Is your claim that all programs are just doing lookup?
> If the response to a algebraic equation was based upon probability of tokens in a corpus...
Ah, I see the confusion. When I say “probability distribution” I do not mean “for each option, an empirical fraction out of all the options, that this particular option appeared in the corpus”. Rather, by “probability distribution”, I mean (in the discrete case) “an assignment of a number which is at least zero and at most one, to each of the options, and such that the sum of the assigned values add up to 1”. I am allowing that this assignment of values is computed (from what is being conditioned on) in any way whatsoever .
If the correct answer is a number, it may compute the entire correct number through some standard means, and then look at however many correct tokens from the number are already present, and assign a probability of 1 to the correct next one, and 0 to all other tokens. If conditioning on a partial answer that has parts wrong, it may use an arbitrary distribution.
Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Model training and evolution are very different processes, but they are both ways of optimizing a physical system. It may be the case that evolution can give rise to intelligence and model training can’t, but we need some argument to prove that.
Yes. Obviously. I can create plans and think on them. I can think without the need for an internal monologue or talking to myself. This has nothing to do with modalities either: I do not think through text. I use text when I do as a way of conveying the thoughts I already have.
Anyone who claims we don't have the ability to form concepts in our head distinct from the medium in which they're transmitted is saying we're effectively ears, eyes, and skin. That the modality is what is important for intelligence.
This is clearly false, and academic silliness aside, yes—emphatically—humans and intelligent agents have internal concepts and models of the world.
Have a look at the OthelloGPT papers (can provide links if you're interested). This is one of the reasons people are so interested in them!
Please do :)
The references in that paper are also good reading!
You're limiting your view of their capabilities on the output format.
> Not so with LLMs!! Generative LLMs do not have a prior concept available before they start emitting text.
How do you establish that? What do you think of othellogpt? That seems to form an internal world model.
> That the "temperature" can chaotically change the output as the tokens proceed
Changing the temperature forcibly makes the model pick words it thinks fit worse. Of course it changes the output. It's like an improv game with someone shouting "CHANGE!".
Let's make two tiny changes.
One, let's tell a model to use the format
<innerthought>askjdhas</innerthought> as the voice in their head, and <speak>blah</speak> for the output.
Second, let's remove temperature and keep it at 0 so we're not playing a game where we force them to choose different words.
Now what remains of the argument?
The "generation" of strings is related to this output format. It's critical to how they work. Legerdemain has been performed to argue that that's irrelevant, and the real intelligence or concepts are sitting inside the network architecture of the trained model prior to generation. But if that were the case, generation could be done based upon the conceptual representation, and not a syntactical representation token by token. This is not currently the case with LLMs.
I'd turn this question around: if my question is irrelevant, how would one go about building an effective real-world LLM that understand concepts and doesn't use likelihood lookups on a token-per-token basis, but instead generated directly from the conceptual basis? Such an argument, if it exists, would make me very happy.
Please note, I understand that there are prior systems which do this. Generative zero-shot transformer models didn't adopt this approach because it's elegant, but because it is efficient to compute with large data sets and has useful efficacy in generating strings. Some are creative. Some are more "accurate." The temperature parameter can affect which of those cases it selects, if any.
People have short memories, but the people who are both appreciative of LLMs as an engineering feat and critical of their claims of intelligence have been saying this for years. They've said that their token likelihood model is effective for seeing things well covered in the data set. They've been saying that due to the sparsity and structure of human language, large scale approximate compression ("curve fits") would be highly effective and efficient. They've been saying that due to the fact these are large scale fits of a data set, the models would eventually converge to something looking like the known knowledge they're trained on, and not exponentially accelerate in knowledge. All of these predictions have proved to be correct or looking highly likely at this point.
Transformer-based LLMs are a neat algorithmic approach to curve fits. But they are curve fits. Things like cosine transforms in JPEGs, wavelet or Fourier reconstruction in CAT scans, audio signal reconstruction from basis functions are also approximate reconstruction models that function along these lines, albeit in a nice Euclidian space without the generative parts of a transformer. But it was precisely knowledge of how systems like that worked which allowed scientists to understand and predict the limitations of these systems a long time ago. Lots of money and fresh eyes have created a useful computation technique, but these insights have been forgotten. I hope—truly—that progress happens in this space. But the critiques stand and there would be lots to gain by a less-hyped acknowledgement of where we are with these models and the tradeoffs baked into them as a compromise for them to be useful.
While sufficiently does a lot of the heavy lifting here, the indistinguishable criteria implicitly means there must be no-way to tell if it is not the real thing. The belief that it is the real thing comes from the intuition that anything that can be everything a person must be, but have that fundamental essence of being a person. I don't think people could really conceive an alternative without resorting to prejudice which they could equally apply to machines or people.
I take the arguments such as in this paper to be instead making the claim that because X cannot be Y you will never be able to make X indistinguishable from Y. It is more a prediction of future failure than a judgment on an existing thing.
I end up looking at some of these complaints from the point of view of my sometimes profession of Game Developer. When I show someone a game in development to playtest they will find a bunch of issues. The vast majority of those issues, not only am I already aware of, but I have a much more detailed perspective of what the problem is and how it might be fixed. I have been seeing the problem, over and over, every day as I work. The problem persists because there are other things to do before fixing the issue, some of which might render the issue redundant anyway.
I feel like a lot of the criticisms of AI are like this they are like the playtesters pointing out issues in the current state where those working on the problems are generally well aware of particular issues and have a variety of solutions in mind that might help.
Clear statements of deficiencies in ability are helpful as a guide to measure future success.
I'm also in the camp that LLM's cannot be an AGI on its own, on the other hand I do think the architecture might be extended to become one. There is an easy out for any criticism to say, "Well, it's not an LLM anymore".
In a way that ends up with a lot of people saying
.The current models cannot do the things we know the current models cannot do
.Future models will not be able to do those things if they are the same as the current ones
.Therefore the things that will be able to do those things will be different
That is true, but hardly enlightening.
I think a lot of people disagree with this. People think if we just keep adding parameters and data, magic will happen. That’s kind of what happened with ChatGPT after all.
For instance LLMs are pretty much stateless without their context window. If you treat the raw generated output as the first and final result then there is very little scope for any advanced consideration of anything.
If you give it a nice long context, give it the ability to edit that context or even access to a key-value function interface, then treat everything it says as internal monologue except for anything in <aloud></aloud> tags which is what the user gets to see. There are plenty of people who see AGI somewhere along that path, but once you take a step down that path, it's no-longer "Just an LLM" the LLM is a component in a greater system.
You could equate saying something "aloud" to calling an API.
It's probably better to have implicitly closed tags rather than requiring a close tag. It would be quite easy for a LLM to miss a close tag and be off in a dreamland.
Possibly addressing comments to the user or itself might allow for considering multiple streams of thought simultaneously. IRC logs would be decent training data for it to figure out many voice multi-conversations (maybe)
Something people don't seem to grasp is that the training data mostly doesn't contain any reasoning. Nobody has published brain activity recordings on the internet, only text written in human language.
People see information, process it internally in their own head which is not subject to any outside authority and then serialize the answer to human language, which is subject to outside authorities.
Think of the inverse. What if school teachers could read the thoughts of their students and punish any student that thinks the wrong thoughts. You would expect the intelligence of the class to rapidly decline.
I wonder if the trend towards using synthetic, AI-generated training data will make it easier to train models that use <aloud> effectively? AI’s could be trained to use reasoning and show their work more than people normally do when posting on the Internet. It’s not going to create information out of nothing, but it will better model the distribution that the researchers want the LLM to have, rather than taking distributions found on the Internet as given.
It’s not a natural distribution anyway. For example, I believe it’s already the case that people train AI with weighted distributions - training more on Wikipedia, for example.
My guess is that the quest for the best training data has only just begun.
There are multiple avenues to train a model to do this. Most simply is a finetune on training examples where the the internal monologue is constructed in a manner that precedes the <aloud> tag and provides additional reasoning before the output.
I think there is also scope for pretraining with a mask to not attempt to predict (or ignore the loss, same thing) certain things in the stream. For example to give time codes into the data stream. The training could then have an awareness of the passing of time but would not generate time codes as a prediction. Time codes could then be injected into the context at inference time and it would be able to use that data.
So I think you’re right, it’s not enlightening. Criticism of overconfident predictions won’t be enlightening if you already believe that they’re overconfident and the future is uncertain. Conversations might be more interesting if not so focused on bad arguments of the other side.
But perhaps such criticism is still useful. How else do you deflate excessive hype or skepticism?
I am continually surprised at how relevant and pervasive one of Kurt Vonnegut’s major insights is: “we are what we pretend to be, so we must be very careful about what we pretend to be”
https://en.wikipedia.org/wiki/Life_imitating_art
Everyone in the "life imitates art, not the other way around" camp (and also neo-platonists/gnostics i.e. https://en.wikipedia.org/wiki/Demiurge ) is getting massively validated by the modern advances in AI right now.
The extension of that argument being made in the paper is that a model trained on language tokens spewed by humans is incapable of actually reaching that limit where this illusion will never breakdown in resolution. That also seems reasonable to me. They use the word "languaging" in verb form as opposed to "language" as a noun to express this.
Language Identification in the Limit in short tells us that if there is an automaton equivalent to human language then, if it's at most a regular automaton it can be identified ("learned") by a number of positive only examples approaching infinity, and if it's above regular then a number of negative examples approaching infinity is also needed to identify it. Chomsky based his "Poverty of the Stimulus" argument about linguistic nativism (the built-in "language faculty" of humans) on this result, known as Gold's Result after Mark E. Gold who proved it in the setting of Inductive Inference in 1964. Gold's result is not controversial, but Chomsky's use of it has seen no end of criticism, many from the computational linguistics community (including people in it that have been great teachers to me, without having ever met me, like Charniak, Manning and Schutze, and Jurafsky and Martin) [1].
Those critics generally argue that human language can be learned like everything and anything else: with enough data drawn from a distribution assumed identical to the true distribution of the data in the concept to be learned, and allowing a finite amount of error with a given probability, i.e. under Probably Approximately Correct Learning assumptions, the learning setting introduced by Leslie Valiant in 1984, that replaced Inductive Inference and that serves as the theoretical basis of modern statistical machine learning, in the rare cases where someone goes looking for one. Around the same time that Valiant was describing PAC-Learning, Vapnik and Chervonenkis were developing their statistical learning theory behind the Iron Curtain and if you take a machine learning course in school you'll learn about the VC Dimension and wonder what's that got to do with AI and LLMs.
The big question is how relevant is all this to a) human language and b) learning human language with an LLM. Is there an automaton that is equivalent to human language? Is human language PAC-learnable (i.e. from a polynomial number of examples)? There must be some literature on this in the linguistics community, possibly the cognitive science community. I don't see these questions asked or answered in machine learning.
Rather, in machine learning people seem to assume that if we throw enough data and compute at a problem it must eventually go away, just like generals of old believed that if they sacrifice enough men in a desperate assault they will eventually take Elevation No. 4975 [2]. That's of course ignoring all the cases in the past where throwing a lot of data and compute at a problem either failed completely -which we usually don't hear anything about because nobody publishes negative results, ever- or gave decidedly mixed results, or hit diminishing returns; as a big example see DeepMind's championing of Deep Reinforcement Learning as an approach to real world autonomous behaviour, based on the success of the approach in virtual environments. To be clear, that hasn't worked out and DeepMind (and everyone else) has so far failed to follow the glory of AlphaGo and kin with a real-world agent.
So in short, yeah, there's a lot to say that we may never have enough data and compute to achieve a good enough approximation of human linguistic ability with a large language model, or something even larger, bigger, stronger, deeper, etc.
__________________
[1] See: https://languagelog.ldc.upenn.edu/myl/ldc/swung-too-far.pdf for a history of the debate.
my limited experience of reading Chalmers is that he doesn't actually present evidence - he goes on a meandering rant and then claims to have proved things that he didn't even cover. it was the most infuriating read of my life, I heavily annotated two chapters and then finally gave up and donated the book.
One thing I appreciate is he often states his premises, or what modern philosophers seem to call "commitments". I wouldn't go so far as to say he uses air-tight logic to reason from these premises/commitments to conclusions - but at the least his reasoning doesn't seem to stray too far from those commitments.
I think it would be fair to argue that not all of his commitments are backed by physical evidence (and perhaps some of them could be argued to go against some physical evidence). And so you are free to reject his commitments and therefore reject his conclusions.
In fact, I think the value of philosophers like Chalmers is less in their specific commitments and conclusions and more in their framing of questions. It can be useful to list out his commitments and find out where you stand on each of them, and then to do your own reasoning using logic to see what conclusions your own set of commitments forces you into.
Like your argument: embodiment is missing in LLMs, but is it needed for AGI? Nobody knows.
I feel we first have to do a better job defining the basics of intelligence, we can then define what it means to be an AGI, and only then can we prove that something is, or is not, AGI.
It seems that we skipped step 1 because its too hard, and jumped straight to step 3.
It's a real mess.
I think the deeper point of the paper is that you simply cannot generate an intelligent entity by just looking at recorded language. You can create a dictionary, and a map - but one must not mistake this map for the territory.
It is also not universally impossible to reconstruct a function of finite complexity from only samples of its inputs and outputs. It is sometimes possible to draw a map that is an exact replica of the territory.
They're not really trying to create a human brain, so far as I can tell. They're trying to create an oracle, by feeding it all existing human utterances. This is certainly not going to succeed, since the truth is not measurable post-facto from these utterances.
The claim regarding reconstructing functions from samples of its ins and outs is false. It's false both mathematically, where "finite complexity" doesn't really even have a rigorous definition - and metaphorically too.
Maps are never the territory.
(Finite complexity is rigorously definable, it's just that the definition is domain-specific).
I like how Chomsky deals with it who doesn't have any spirituality at all, the big degenerate materialist:
As far as I can see all of this [he's speaking about the Loebner Prize and the Turing test in general] is entirely pointless. It's like asking how we can determine empirically whether an aeroplane can fly the answer being if it can fool someone into thinking that it's an eagle under some conditions.
https://youtu.be/0hzCOsQJ8Sc?si=MUXpmIwAzcla9lvK&t=2052
(My transcript)
He's right, you know. It should be possible to tell whether something is intelligent just as easily as it is to say that something is flying. If there are endless arguments about it, then it's probably not intelligent (yet). Conversely, if everyone can agree it is intelligent then it probably is.
Because it's not easy to tell whether something is flying. Definitions like that fall apart every time we encounter something out of the ordinary. If you take the criterion of "there's no discussion about it", then you're limiting the definition to that which is familiar, not that which is interesting.
Is an ekranoplan flying? Is an orbiting spaceship flying? Is a hovercraft flying? Is a chicken flapping its wings over a fence flying?
Your criterion would suggest the answer of "no" to any of those cases, even though those cover much of the same use cases as flying, and possibly some new, more interesting ones.
And I don't think an AGI must be limited to the familiar notion of intelligence to be considered an AGI, or, at the very least, to open up avenues that were closed before.
For example, you can't miss the fact that a five-year old child is intelligent and you can't miss the fact that a stone is not. There may be all sorts of things in between for which we can't be sure, or whose intelligence depends on definition, or point of view, etc. but when something is intelligent then it should leave us no doubt that it is. Or, if you want to see it this way: if something is as intelligent as a five-year old child then it should leave us no doubt that it is.
I'm basically arguing for placing the bar high enough that when it is passed, we can be fairly certain we're not mistaken.
>> I can't disagree more. Or maybe I actually agree.
I find myself in that disposition often :)
I strongly disagree. There are many reasons we might not recognize its intelligence, such as:
- it operates on a different timescale than we do.
- it operates at a different size scale than we do.
- we don't understand its language, its methods, or its goals
- Cartesian-like ideological blindness ("only humans have experience, all other things are automata, no matter how much they seem otherwise")
Throughout human history, certain people have even managed to doubt the intelligence of other groups of humans.
But why should current debate have anything to do with all those situations? By analogy, suppose there exists a race of alien birds on a faraway planet that can fly in a way that we wouldn't recognise as flying. Perhaps they have an antigravity gland or poop exotic matter that sticks to their butts and lets them move around without touching the ground. Does that affect our ability to recognise flight when we see it on Earth? I don't think so. When you see e.g. an eagle, fly, you know it's flying, regardless of whether anything else is, or might be conceived as, flying, or not.
Equally, does the possibility of an alien intelligence like no intelligence we have ever seen before make any difference to how we recognise intelligence here on Earth and right now? The debate is about the intelligence of computers. Should we consider the possibility that computers are about to develop an alien kind of intelligence like no intelligence we have ever seen? I know there's such a current of thought in various circles but it doesn't seem to be based on anything but wishful thinking (or dreadful thinking?).
As to "Cartesian blindness" and racism etc - science is self-correcting. Even if we spend some time confused about what is and isn't intelligence, we get there in the end. The question is how we identify intelligence in the here and now, with what we know and understand about the world in the present.
We already have alien intelligences on this planet that are not often getting invoked in discussions of the possible unfamiliarity of the difference of computers. Let's take the octopus distributed thinking or the hive's collective intelligence. They could shine a light on how computers do or don't work, yet how many people are holding the torch?
I think self-correcting of science can be nice, but the acknowledge of racism operates on cultural scales of decades and centuries. The corrected discussions are likely to appear long after computer thinking breakthroughs which happen on the scale of years to decades. So you're right, but that doesn't matter today. But I admit I'm not sure if that's what you meant here.
Scant few, unfortunately. I think in the AI community there is a general agreement that bees and ants have some kind of intelligence but I don't see anyone doing much about it, like using it as a model for AI (although note e.g. Ant Colony Optimisation and other biologically inspired algorithms). For example, I have seriously considered applying for funding for a project titled "Arthropod-Like Intelligence" that would seek to create an artificial agent (a robot of some sort) with autonomous capabilities at the level of a spider. Unfortunately, every time I start to write up the proposal I immediately imagine the ridicule it -and I- would be subjected to by any funding committee and fellow researchers in AI and I give up. So yes, the current agreement about what counts as intelligence is limiting to the advancement of science. But there is some progress, slow as it is- thanks to people bolder than myself.
With octopi I think it's a different matter. People generally don't get to see how octopi behave. Once they're shown examples, like in videos etc, I think most are convinced. So it's more an element of surprise, rather than a real resistance to the idea. I think the debate is more on different aspects of let's say broader cognition, like self-awareness, the ability to feel pain, etc. Again I think the safe bet here is to adopt the default position that all animals are intelligent, self-aware and can feel pain, since there's no reason that humans should be special in that respect. But, really, it's not my field so my opinion doesn't matter in the grand scheme of things.
>> The corrected discussions are likely to appear long after computer thinking breakthroughs which happen on the scale of years to decades.
Yeah, unfortunately science takes a very long time. But it works in the end and there's no better way we know. I don't guess I have to argue about that though.
Not that long ago, a whole lot of humans (the majority in some continents) asserted other a group of other humans were not intelligent so strongly they purchased them as property and treated them worse than even farm animals, so i think you can basically throw this one out the window.
What probably happened, and still happens, is that some people underestimate the intelligence of other people and think they're not as intelligent as themselves, not that they don't have intelligence.
Thomas Jefferson who generally seemed to oppose slavery says this in his book, "Notes on the State of Virginia" (1785):
"In general, their existence appears to participate more of sensation than reflection."
"Comparing them by their faculties of memory, reason, and imagination, it appears to me that in memory they are equal to the whites; in reason much inferior, as I think one could scarcely be found capable of tracing and comprehending the investigations of Euclid; and that in imagination they are dull, tasteless, and anomalous."
Doesn't even sound like he's talking about the same species. We can scarcely agree on the intelligence of other humans. Let's not even get into the topic of animals.
So the idea that intelligence will manifest and we'll all just see it and agree...Yeah No.
He's saying they're of inferior "reasoning" (but equal in memory). That's hardly saying they're not intelligent or recognising them as not intelligent.
I don't think you will find anyone saying what you think someone's saying. "Not intelligent" is indeed a rock or a piece of wood. "Inferior" is a different concept.
"Inferior", "sub-human", "feeble-minded", sure, people keep saying things like that for other humans. But, "not intelligent" in the sense of "can't use language", "can't use tools", "can't tie own shoelaces", that would be very hard to maintain in the face of very easy to make observations. Which, you know, is exactly my point.
Are you using "intelligent" to mean "very smart" or something similar? That's not what I'm saying. I'm really just pointing to the difference between 0 intelligence, like a rock, and undeniable intelligence, like a 5-year old child.
Ok i guess the argument then...So what are these obvious signs of Intelligence the machines in question are not displaying?
From the few you gave as example,
"can't use language" - Obviously not a problem
"can't use tools" - Digital tools, Some physical tools is certainly possible today
"can't tie own shoelaces" - Don't know any animal that can do this and animals pass your bar of intelligence.
Sure we're having endless arguments about the Intelligence of Sota LLMs today but almost none of it has anything to do with observable, testable capabilities.
In other words, the problem of these debates isn't that people are disagreeing seeing both the bird and plane have willful sustained airtime (aka flying). It's that they've seen both and decided for entirely arbitrary reasons to label one, "real flying" and the other, "fake flying".
And if you then ask the reasonable question, "Which properties separate the so called 'real flight' from 'fake flight' and how do we test for it?", then nobody seems to have any clue.
To summarise my argument again: if we can all agree that something is intelligent, then it probably is. If we can't, then it probably isn't, especially, I would add, if there is substantial disagreement.
One motivation for this is to avoid an unending quest for the right definition. Once we can all agree what (artificial) intelligence is, it will be much easier to agree to a definition. But while we're all looking at the same thing and can't agree on what it is, how can we agree on a definition, and then use it to support one or the other side of the debate?
Take human intelligence. I don't think anyone seriously doubts that humans are intelligent, except perhaps for the purpose of being contrary, or being a philosopher[1]. We may not know what "intelligent" means, but we can agree that whatever it is, it's something that humans have. Yes, that's an arbitrary decision, but as long as we can agree on it, it doesn't matter that it's arbitrary: it matters that there is consensus, based on common experience, and we all know what we're talking about. We can't get to a definition of a phenomenon before we all agree that it is there: we all agreed that fire is fire, water is water and ice is ice, a long time before we had any sort of commonly agreed definitions of them.
"I can't put my finger on it but I know it when I see it"- that's a very simple way to avoid discussions that lead nowhere.
And there are way too many discussion that lead nowhere in AI.
Edit: btw, in research the discussions are much more focused. E.g. "LLMs can plan": that's a concrete, testable claim. No reason to define intelligence to resolve it. Much more progress is made this way than by endlessly chasing our tails about what is or isn't intelligent.
___________
[1] The purpose of science is to pose questions and answer them. The purpose of philosophy is to pose questions and question them.
No, I think we need a better heuristic than consensus. The consequences of a bad heuristics are, at best, wasting time, but at worst, pulling the trigger on something heinous because it was insufficiently understood.
I agree that there are dangers in trusting consensus as a mechanism for advancement of knowledge, and I see your point about risking losing focus on the border. But I think it is also very important to have a common body of knowledge that we can all agree on. Take flying again: nobody disputes the fact that planes are flying (again, nobody who isn't just being contrary). We can work out the gray areas in time and they don't necessarily make progress impossible. E.g. maybe we can't agree on whether hovercrafts fly, but we can still make them and use them.
So there is a limiting effect, like I point out in my other comment, but it is not a hard lock on progress. And I think that avoiding endless philosophical discussions is a big advantage.
If there is a better heuristic then I'm happy to embrace it btw. But, is there?
There's a paper discussing heuristics for determining the moral worth (not intelligence) of being which advocates the precautionary principle: http://www.faculty.ucr.edu/~eschwitz/SchwitzPapers/AIRights2...
> avoid acting heinously by the standards of any reasonable ethical principle that draws a significant proportion of well-informed, thoughtful theorists
Is it a problem though? Their existence are unrelated to how we categorize them.
That matters only in communication. “if everybody agrees” lowers/removes the risk of miscommunication.
If “hovercraft is flying” for you, but not for 50% the world, it makes it somewhat more difficult to communicate. (Easily solved with some qualifications, but that requires admission the questionability of “hovercraft is flying”)
> you're limiting the definition to that which is familiar, not that which is interesting.
You made an Interesting point - good food for thought.
Counterpoint: It seems natural and useful that only similar things get to use same word.
> And I don't think an AGI must be limited …
Could you expand on why does it matter and what would be impacted by such lenient (or strict) classification?
In the same vein: https://eschwitz.substack.com/p/strange-intelligence-strange...
True. I doubt that field experts are directly affected by the naming, but indirect effect might come via less knowledgable (AI wise) financial decision makers.
I see a risk that those decision makers (and society) would be mislead if they were promised AGI (based on their “strict” understanding, what’s in the movies), but received AGI (based on “relaxed” meaning). Informed consent is usually good.
Though surely that can be resolved with more public discourse; maybe “relaxed” version will become the default expectation.
I guess this also disproves Minsky's 'Society of Mind' conjecture - a large part of human cognition (System 1) does not require the complex interaction of heterogeneous mental components.
The same way they "show" sadness or contrition or excitement?
We need to be careful with our phrasing here: LLMs can be prompted to provide you associated phrases that usually seem to fit with the rest of the word-soup, but whether the model is actually demonstrating "logical thinking" or "self-correction" is a Chinese Room problem [0]. (Or else a "No, it doesn't, I can tell because I checked the code.")
Beyond that fact that human minds created them, I doubt that LLMs can tell us anything about the abilities of the human mind or what language in humans requires.
The most we can learn from LLMs about ourselves will be in how we react to them, or more broadly, what the datasets LLMs use show about who we are.
My own thought on this (as someone who believes embodiment is essential) is to consider the rebuttals to Searle's Chinese Room thought experiment.
For now (and the foreseeable future) humans are the embodiment of LLMs. In some sense, we could be seen as playing the role of a centralized AIs nervous system.
Not much new here. The basic criticism is that LLMs are not embodied; they have no interaction with the real world. The same criticism can be applied to most office work.
Useful insight: "We (humans) are always doing more than one thing." This is in the sense of language output having goals for the speaker, not just delivering information. This is related to the problem of LLMs losing the thread of a conversation. Probably the only reasonably new concept in this paper.
Standard rant: "Humans are not brains that exist in a vat..."
"LLMs ... have nothing at stake." Arguable, in that some LLMs are trained using punishment. Which seems to have strong side effects. The undesirable behavior is suppressed, but so is much other behavior. That's rather human-like.
"LLMs Don’t Algospeak". The author means using word choices to get past dumb censorship algorithms. That's probably do-able, if anybody cares.
The converse is that people are "just" statistical distributions of the signals produced by them but I don't know if there are people who claim they are nothing more than statistical distributions.
I think people are confused because they do not really understand how software and computers work. I'd say they should learn some computability theory to gain some clarity but I doubt they'd listen.
It's pretty clear that these optimisation processes lead to emergent behaviour, both in ML and in the natural sciences. Computability theory isn't really relevant here.
That is patently false - most of those things are firmly in the realm of engineering, especially these days. Mathematics is good for grounding intuition though. But why is this relevant to the OP?
I'm not equating anything here, just pointing out that the fact that AI runs in software isn't a knockdown argument against anything. And computability theory certainly has nothing useful to say in that regard.
Sidenote: the authors make a mistake when citing Wittgenstein to find similarity between humans and LLMs. Language modelling on a static dataset is mostly not a language game (see Bender and Koller's section on distributional semantics and caveats on learning meaning from "control codes")
IIRC DPO doesn’t have human feedback in the loop
Does this have a specific name?
Conversations you have with like chatgpt are likely stored, then sorted through somehow, then added to an ever growing dataset of conversations that would be used to train entirely new models.
"Here is what we think about this current hot topic; please read our stuff and cite generously ..."
> Language completeness assumes that a distinct and complete thing such as `a natural language' exists, the essential characteristics of which can be effectively and comprehensively modelled by an LLM
Replace "LLM" by "linguistics". Same thing.
> The assumption of data completeness relies on the belief that a language can be quantified and wholly captured by data.
That's all that a baby has, who becomes a native speaker of their surrounding language. Language acquisition does not imply totality of data. Not every native speaker recognizes exactly the same vocabulary and exactly the same set of grammar rules.
But anyway I agree with you. This is just a confused HN comment in paper form.
Feedback and interaction is not vital for acquisition for secondary language learning at least according to one theory.
And if that’s good enough for adults it might be good enough for sponge-brain babies.
You should publish your critique of their research in that same journal.
P.s. if you find any grave mistakes, you can contact the editor in chief, who happens to be a linguist.
No thanks; that would be at least twice removed from Making Stuff.
(Once removed is writing about Making Stuff.)
Their critique is written here, in plain english. Any fault with it you can just mention. The "I won't read your comment unless you get X journal to publish it" seems really counterproductive. Presumably even the great Journal of Language Sciences is not above making mistakes or publishing things that are not perfect.
I read it as a clear refutation of the assertion that the authors "have no connection" to AI (or to AI hype; unclear from the OP).
Btw, the OP is a typical ad-hominem, drawing attention to who is speaking rather than what they're saying.
Are there people who ride the hype wave of AI? Sure.
But how can you tell from where you sit? How do you come to such a judgment? Are you being thoughtful and rational?
Have you considered an alternative explanation? I think the odds are much greater that the authors’ academic roots/training is at odds with what you think is productive. (This is what I think, BTW. I found the paper to be a waste of my time. Perhaps others can get value from it?)
But I don’t pretend to know the authors’ motivations, nor will I cast aspersions on them.
When one casts shade on a person like the comment above did, one invites and deserves this level of criticism.
> ...we argue that it is possible to offer generous interpretations of some aspects of LLM engineering to find parallels with human language learning. However, in the majority of key aspects of language learning and use, most specifically in the various kinds of linguistic agency exhibited by human beings, these small apparent comparisons do little to balance what are much more deep-rooted contrasts.
Now, why is this so hard to stomach? This is the argument of this paper. To feel like this extremely general claim is something you have to argue against means you believe in a fundamental similarity between what our linguistic agency and the model. But is embodied human agency something that you really need the LLMs to have right now? Why? What are the stakes here? The ones actually related to the argument at hand?
This ultimately not that strong of a claim! To the point that its almost vacuous... Of course the LLM will never learn the stove is "hot" like you did when you were a curious child. How can this still be too much to admit for someone? What is lost?
It makes me feel little crazy here that people constantly jump over the text at hand whenever something gets a little too philosophical, and the arguments become long pseudo-theories that aren't relevant to argument.
Another team, with a similar role in a different part of the org has jumped (feet first) into optimizing large language models to turn them into agents, without consulting the business about whether they need such things. RAG, LoRA and all this optimization is well and good, but this engineering focus has found no actual application, expect wasting several million bucks hiring staff to do something nobody wants.
I don't think such an individual is possible.
I'm struggling to articulate my cognitive dissonance here, but is there any empirical evidence that LLMs, or their underlying machine learning technology, share anything at all with biological consciousness beyond a convenient metaphor for describing "neural networks" using terms borrowed from neuroscience? I don't know that it necessarily follows that just because something was inspired by, or is somehow mimicking, the structure of the brain and its basic elements, that it should necessarily relate to its modeled reality in any literal way, let alone provide a sufficient basis for instantiating a phenomena we frankly know very little about. Not for nothing, but our models naturally cannot replicate any biological functions we do not fully understand. We haven't managed to reproduce biological tissues that are exponentially less complex than the brain, are we really claiming that we're just jumping straight past lab-grown t-bones to intelligent minds?
I'm sure most of the people reading this will have seen Matt Parker's videos where they "teach" matchbooks to win a game against humans. Is anyone suggesting those matchbooks, given infinite time and repetition, would eventually spark emergent consciousness?
> The argument would be that that conceptual model is encoded in the intermediate-layer parameters of the model, in a different but analogous way to how it's encoded in the graph and chemical structure of your neurons.
Sorry if I have misinterpreted anyone. I honestly thought all the "neuron" and "synapse" references were handy metaphors to explain otherwise complex computations that resemble this conceptual idea of how our brains work. But it reads a lot like some of the folks in this thread believe it's much more than metaphors, but rather a literal analog.
Notice how LeCun is the only researcher being honest about this in a public fashion. Meta is committed to AI already and will at least match the spend of competitors anyway, so he doesn't have as much pressure to try and convince investors that this rabbit whole is deeper.
Don't get me wrong, LLMs are a tremendous improvement on knowledge compression and distillation, but it's still unreliable enough that old school search is likely a superior method nonetheless.
Look at the arc of NLP. Large language models fit the pattern. One could even say that their development (next token prediction with a powerful function approximator) is obvious in hindsight.
However, given the hype cycle, combined with broad levels of ignorance of how LLMs work, it is an open question if even amazing progress will impress people anymore.
The value is lower than perceived because there is an assumption that what's preventing higher value delivery from the investment is for the models to get better at generating responses from prompts. But there are two issues with this position.
1. LLMs still require plenty of assistance where they are writing production ready code and making function-calls, especially if the original API wasn't designed with LLMs in mind. Unless there is a leap in architecture that's going to make all the tools we have, including non-API ones easily accessible by the models to interact with, the amount of glue code required to make it all work increases the potential for features that users can use, but not necessarily deliver higher value to developers in that a lay person still probably can't develop software with an LLM copilot in toe. So yes, no one is going to be impressed even if we see models improve further on benchmarking.
2. Long-Horizon goals. Long-term research or even project management requires interdisciplinary understanding of how all the goals around success relate to each other and most importantly how to assess if an outcome is leading to a goal accomplishment or not. There isn't an architectural foundation for the models to be grounded in a reality that presupposes these abilities.
What I fail to see, is how improving next token prediction will materially move the needle on these other aspects of intelligence that aren't necessarily related to generating an output or a series of outputs orchestrated over an evolving set of requirements.
Honestly lI think that the LLM portion of the human brain has likely be surpassed by existing models
With very careful discussion, there are some really interesting concepts in play. This paper however does not strike me as worth most people’s time. Especially not regarding consciousness.
I am a fan of this paper, warts and all ! (and the paper summary paragraph contained some atrocious grammar btw)