Hacker News Reader

> LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition.

If this claim of yours was sustainable, your grievance with the writing style would make a lot of sense. But GPT-4 can clearly reason about novel problems which were not in its training data. I think you should consider reading some of the many citations in the document which describe examples of that happening.

It doesn't reason, it produces output that better conforms to your expectations of a reasoning actor.

The models are showing emergent behavior. You can't say with certainty that they aren't able to reason, just because they were not designed to do so.

I can confidently say that you do not have a theory of rationality which explains the workings of an LLM.

On that point we agree.

If it quacks like a duck..

Really, if it always produces output that conforms to "expectations of a reasoning actor", isn't that defacto a "reasoning actor"? What's the use of making the distinction?

Because of all the ways in which it's obviously not a reasoning actor, starting from the design of the system, and all the ways in which it is literally engineered, by design, to output text that sounds like a human wrote it, based on consuming the Internet.

In all ways except .. its output? Which is the only thing that matters? What if you were engineered? Would that invalidate your abilities?

I apologize for being difficult.

> In all ways except .. its output? Which is the only thing that matters?

Who decided that this was the only thing that matters?

The thing that concerns me isn't that it answers questions in a way that resembles a human. What concerns me is that some people think it thinks like a human, which it does not, and writing about it in an anthropomorphic way exacerbates this problem.

The issue at hand is not the observable output of the system, but the way in which the system works — which we have knowledge of. It's not magic, it's not emergent, it's not sentient, and it's certainly not intelligent. It's a (very fancy and impressive!) predictive text system. Imbuing it with extra properties is disingenuous.

My rationality and agency is not a question of you interpreting the words and letters that appear in a textbox on a screen. The same is not true of a large language model. You do not have a rigorous theory of rationality and agency with which to interrogate a large language model.

LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this board state is used, because if you patch activation so that board state is modified, the model generates moves for this modified board state.

> LLMs clearly develop internal representations, this is an empirical fact.

you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems.

From your citation itself:

> More broadly, it would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state?

Okay, I think we are in agreement. Let's say current LLM architectures are clearly capable of developing internal representations, not just learning surface statistics, and it can execute algorithmic computation as complex as computing Othello board states, and it can completely generalize out of distribution thanks to such algorithmic computation. (One experiment was to completely eliminate any Othello games starting with move C5 from the training data. The model had no trouble generating only legal moves for C5 games.)

I agree whether LLMs trained on natural language do develop world model is an interesting and open empirical question. They clearly can, but we don't know whether they do. I do consider the evidence pretty suggestive that they do. But this is clearly just an empirical question, not a question that can be answered by arguments as useless as "they are statistical models", because statistical models can and do learn world model.

I actually consider your considering this result not incredibly surprising pretty surprising.

The othello example explicitly was trained on the rules of the game - it isn't suprising that you can find a representation of game state.

LLMs are trained on the "rules" of human written language, which are sufficiently distinct from anything in the actual world that find world modelling would, indeed, be a surprise to me.

> I hope the author can clarify their choice of phrasing in their work

From TFA's conclusions section:

"Open debates over whether we describe LLMs as understanding language, and whether to describe their actions using agency-related words like know or try, are largely separate from the questions that I discuss here (Bender & Koller, 2020; Michael, 2020; Potts, 2020). We can evaluate whether systems are effective or ineffective, reliable or unreliable, interpretable or uninterpretable, and improving quickly or slowly, regardless of whether they are underlyingly humanlike in the sense that these words evoke."

>LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models.

As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Blake Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics.

To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so.

I found this recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0>

>Anthropomorphizing them is not just technically incorrect, but morally disingenuous.

Wat

I didn't say it's "not real AI", but of course whether I meant that or not comes down to a definition of "AI".

In academia (where I am, though my specialty is a step removed), "AI" is exactly the region of research concerned with statistical problem-solving. Machine learning is sometimes synonymous, though I haven't been able to determine whether this is consistent among all self-described AI researchers or just some of them.

Systems like GPTs are not capable of "thought", full stop. There is no ambiguity. If this is hard for you to accept, I'm sorry, but it is a fact. They have no agency.

I saw someone sketch a good explanation. They were responding to a claim that Chat GPT could "pass the Harvard admissions process". While it is true that the system can access sufficient data to respond accurately to questions on an exam, and it could be prompted to generate an essay, it cannot of its own volition choose to submit an application. It doesn't even "know" that Harvard is a real place to which it can apply, nor does it "understand" how it could go about "learning" how to find out about these things it does not know. It simply isn't capable of these things. (And even if it were that wouldn't necessarily be evidence of intelligence, but for now it's a sufficient distinguisher.)

> Wat

Can you explain what about my last point was "wat"-worthy? Do you disagree that it is morally problematic to pretend a thing is sentient when it is in fact not?

So the AI Ethics peeps are the True Scotsmen?

We may just live in a world where there’s room for more than one viewpoint and people are able to research different aspects of the same thing without getting into debates over dogma.

I mean, inclusion -> https://news.ycombinator.com/item?id=34698769

> We may just live in a world where there’s room for more than one viewpoint

AI safety is not a "viewpoint", it is a marketing strategy. The goal of the people leading that charge is to drum up FUD to increase investment in their useless snake oil companies. I feel no remorse for not being inclusive of their "perspective".

It's been a long time since anyone reminded me of Richard Bach's 'illusions'

Thank you for a pleasant break!

https://www.patheos.com/blogs/wakeupcall/2020/05/10-thought-...

It's always a happy moment when someone recognizes my online namesake. Cheers!

At what point are we allowed to use the "reason" or "learn" for AI then? Can we use these terms for non-human animals? If so, which ones? I think LLMs are at the point where adopting an "intentional stance" (using Daniel Dennett's term) is reasonable.

> At what point are we allowed to use the "reason" or "learn" for AI then?

Maybe when the technology is physically capable of thinking for itself, rather than just being statistically likely to respond in a way that appears like the way a human would respond. Which may never happen, for what it's worth.

> I think LLMs are at the point where adopting an "intentional stance" (using Daniel Dennett's term) is reasonable.

I hadn't heard of this term so I looked into it, and wow, what a biased Wikipedia article it has. It's written with a completely partial stance. I've no idea whether to take it seriously, to be honest.

That said, I disagree strongly. LLMs do not "intend" anything, and pretending they do is (in my view) highly problematic. They're just (sophisticated!) prediction engines; nothing more.

> LLMs do not "reason"

What definition of reasoning do you have in mind such that LLMs don't do it?

While llm don't learn, they have now enough token input space and handle context well enough to manage a cyclic conversation.

While not learning, they can still run inferences on what they are fed as context within the token space. And because they are becoming so good at getting context out of structured data, they can use the context provided as a layer on top of their knowledge.

So you can at a very good approximation upload knowledge into them, and such knowledge can represent how to use tools to fetch data they don't know about - i.e. what langchain agents are doing.

Now of course that is not 'llm' some, is llm + code for informaron discovery, but that code is entirely supported and enabled by the existence of good enough llm

What does “reason” mean? It seems like it does everything I expect from something that reasons.

People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out.

Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.

> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.

Google gives the following definition of the verb "reason":

> think, understand, and form judgments by a process of logic.

LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more.

> LLMs obviously reason and understand by any evaluation that can be carried out.

Uh-huh. Sure, Jan.

>LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more.

"LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up.

They don't form judgement or conclusions? Sure looks like they do. So what's the difference ?

What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning.

Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference.

>Uh-huh. Sure, Jan.

Yeah. https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.

This is the problem with non-operational definitions, because now we need to know how you define "think" and "understand" and "form judgments", to move on.

Instead, could you operationally define "reason" in a way that a human is, say, 90 % likely to pass the test and GPT is 10 % likely to do?

Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022.

https://twitter.com/fchollet/status/1631699463524986880

https://github.com/fchollet/ARC

Wow, this is a much more interesting answer than I expected. Thank you!

It is. Although I'd be careful with the conclusions.

The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.

For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196

This seems more like an issue with the tokenizers, no? This doesn't seem more difficult than other problems it solves.

You're right that LLMs can solve abstract analogical reasoning problems https://arxiv.org/abs/2212.09196

Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the equivalent of an artificial visual cortex. We don't have that yet

New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.

… and then perform a careful search of books and the whole internet to be sure what you think is novel hasn’t been thoroughly debated somewhere on stackexchange.

If it's a stochastic parrot, then merely randomizing proper nouns and filler text should be enough to prevent its abstraction ability.

If you're saying that we can't use a problem if any analog of that problem has ever been described, you seem to be arguing more strongly that it is a general intelligence than I am.

I’m saying that people in my circle have been asking what they think are novel questions and getting interesting answers, only to find out that very similar content exists on websites we know are in the training set.

That’s not intelligence that’s computers having better memory than humans. Useful, certainly, but hardly skynet.

I don't think you're being clear about whether the questions were novel. If you discover your question was uncreative, surely e.g. some details, wording, facts, names, or numbers inside the question can be changed to defeat a model that is answering it from memory?

If you're saying that it is not possible to change the details enough to avoid the model being able to answer that type of question, I think you are admitting that the model has learned a generalized ability to answer questions of that class, and is not actually using its memory to answer at all.

I don't care about whether it learned that generalized ability from seeing examples of the question and answer, which it then deduced an algorithm for and generalized -- that's how most people learn most things.

The asker thought they were. They were not. The internet is big and human memories are not.

As an aside, I’m really starting to hate these threads on here, people are constantly reading words that aren’t there in search of gotcha-it’s-skynet. It’s not. It’s just pattern matching and randomness with a giant amount of information encoded.

> As an aside, I’m really starting to hate these threads on here.

I'm not sure what to say, other than that if you'd like to have less frustrating conversations, you could do better than showing up with hearsay where someone asked a question they thought was unique, but it wasn't, and it can't be modified to be unique and then asked again, and you aren't willing to tell us what it was, and possibly don't know yourself.

It is not possible to have a serious conversation about your claim, and that's not because it is being intentionally misunderstood.

> skynet

You're the only person mentioning skynet. The conversation is about a ridiculous claim made up-thread that GPT-4 cannot reason or understand anything, which is disprovable within a few minutes of using it thoughtfully.

This is the exact type of unserious wish-reading I’m talking about. Do better.

You know, François Chollet literally did this, and people don't listen. People should listen to Chollet more.

https://twitter.com/fchollet/status/1638643323748618240

https://arxiv.org/abs/1911.01547

Is ARC a benchmark that GPT-4 can be tested against today? I would be curious to see its results.

Yes it is, and it was tested. GPT-4 can't solve any of the tests.

Yes. However,

The vision problems will require something much more than an image to text objective task. It will require the equivalent of an artificial visual cortex. We don't have that yet.

For abstract analogical reasoning, LLMs don't have a problem with that. https://arxiv.org/abs/2212.09196

That's not stuff an LLM can't do. It's just presented in a way that makes it difficult to do so.

First the vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.

For the text, LLMs don't really have any problem with analogical reasoning https://arxiv.org/abs/2212.09196

We're survival machines, nothing more...

Complex behavior arises from simple systems all the time. You can't prove that these systems don't reason, no matter how loudly thou doth protest.

from the beginning of time people have been overestimating the complexity of things like the human brain and attributing it to magical things (like a creator) far beyond our comprehension but what seems to be happening now is that some people are underestimating it.

I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction.

The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow.

An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know

This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.

So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure.

vvv

But, coming at this as *respectfully and curiously* as I can here:

^^^

I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source.

Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem.

So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.

> something that I know is only working from words as a source.

Your intuition about the architecture might be causing the distrust. Yes, the ends of the model are words in and words out (although GPT-4 is now multi-modal and can accept images as input too), but that doesn't mean the calculations in the middle are dealing in words. The "language" in "large language model" is misleading. It's a large generalized model that imports and exports language.

That is totally possible, the explosion of stuff around this space has kind of blindsided me, so I don’t have much intuition for precisely how these models are built.

It all still feels rather “attribute magic to complexity” to say that there’s some human element skimmed off the patterns of our language, hidden in the coefficients somewhere.

I’m also not a magician, so really who am I to say :)

Does it really provide the statistically most common next token? I don’t think that’s even true. And we’ve seen what looks like emergent behavior as parameters increase. I suspect it is actually creating abstractions at a deeper level than standard DNNs.

I’d be curious to know! I get skeptical when it starts moving into the “emergent properties of language” but language was built by fleshy neurons, it could be that some inherent pattern emerges in the language we speak that reflects the inner workings of our brain, which by trying to emulate makes a similar brain on the other end.

However, that feels a bit magical to me at the moment. Still interesting to be along for the ride humanity is going on!

When you hear someone speak, are you thinking their thoughts?

We are talking about a model of language and our expectations for what the words would look like if a person were using language to reason with. People are confusing one thing (their own interpretation of what they are reading) with another (a statistical model that is being driven to align with certain specific human expectations).