Mechanistic interpretability researchers applying causality theory to LLMs | Hacker News Reader

Mechanistic interpretability researchers applying causality theory to LLMs

cacm.acm.org

117 pointsadunk86 comments

One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights.

So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.

And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.

(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)

You dont have to understand chemistry to be a good cook tho.

there's a 2MP about the related paper: https://www.youtube.com/watch?v=l72ufA-4SzE

This article is not about "reasoning" in the abstract, philosophical sense but is talking about "mechanistic interpretability" research. The title is more like, "can we understand if the 'knowledge' encoded into a neural networks actually corresponds to reasoning-like concepts" and doing that with actual experiments like tweaking weights and activations.

There's an interesting example where researchers saw a model approached clock time calculations and calendar month-day calculations using the same methodology. So then is this because an underlying concept of "cyclical measures" has emerged in the network?

Thanks - I've attempted to put that in the title above, in the hope of representing the article accurately.

(The trouble with a baity title like "Can We Understand How Large Language Models Reason?" is that it generates a barrage of shallow, reflexive responses having little to do with the article. What we want on HN are curious, reflexive responses instead - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....)

I also calculate six months from August as 8+6=14mod12. I wonder if anyone does it differently, this seems like the most plausible technique.

To calculate +/- 3/6/9 months I shift by seasons. 3rd month of summer becomes 3rd month of winter.

That works well cause all months live in a primitive memory palace in my head: an analogue clock face with July at 12 and January at 6. So shifting by 6 means rotating the clock hand from 11 to 5 and immediately visualising what month it falls on.

This might sound inefficient to an LLM but human brains had image processing before language.

For me: January = up, April = left, July = down, October = right.

Never sure why I did this association, maybe it comes from a drawing in a book I read when I was six or somthn?

I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a hallucination. But those weights do not exist for the sake of it, they encode a lot of text the model has seen during training, and I would imagine this is what drives the reasoning. Can you evaluate the following polynomial ... will be related to To evaluate a polynomial ... seen in the training data. This is the level at which I would look for the reasoning, memorized patterns how to do specific things, maybe with some kind of placeholder variables for generalization. Ultimately such a structure would of course also be represented in the weights but I could imagine that this makes it unnecessary hard to understand. Or maybe not, maybe the learned patterns are so complex that they do not have a simple representation.

> In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights.

What is an output token distribution except a set of weights?

I also suspect the learned patterns are not necessarily efficient, though might be by accident.

One could “learn” addition by memorizing a truth table instead of understanding the concept… The truth table itself wouldn’t have much meaning.

>“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.”

what is the basis for this optimism ?

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

no they havent . success so far is totally meaningless and doesn't imply any sort of upward slope .

The researchers in the field disagree with you. Look at conferences like NeurIPS and ICLR to see a steady stream of incremental progress in this area.

trust the researchers bro

What would real progress look like?

Original HN title: "Can We Understand How Large Language Models Reason?"

[stub for offtopicness]

[[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase, and we're trying for something more substantive here.]]

Clickbait article title.

The article body does not presume they reason.

We've edited the title now in the hope of nudging the discussion in a more substantive direction.

No, talking to itself is not reasoning.

Do LLMs have Qualia?

Do people?

Yes.

How do you know?

Because there’s no evidentiary reason to deny the existence of my own phenomenal experience. I appear to experience, and the simplest conclusion to draw from that appearance is that I truly do experience.

I completely agree with that argument, but that only covers one person.

Because I'd be sad if we didn't.

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and you can see life near the end, how the delta of all the same movies opens up into wildly different endings.

To advance further it would need the ability to abstract away the general situation shape and pattern recognize similar situations.

When a mathematician reads a hundred-year-old math paper, they are reproducing in their head the reasoning of someone who died long ago. That is, reasoning can be written down and replicated.

If that works, I think it's fair to say that LLM's are inanimate processes that can generate real reasoning. You can tell when you read it and it makes sense.

There are likely some kinds of reasoning that can't be written down, as well as other forms of understanding, but they also don't replicate nearly as easily.

There is a streamer who plays Diablo 2 by listening to the AI advice and it is quite funny since it is pretty clear that most of the advice is an amalgamation of random, often incorrect advicem

I wonder if it is the same for programming or not, but I vibe coded an android app just to see if I can and it just works. It required a lot of "build the code and correct the errors" pushing though. For example requested code in kotlin but received something else.

As somebody who uses Claude heavily and heavily plays D2R it’s clear he wasn’t using Claude opus…… maybe Haiku or something. Opus isn’t as brain dead as what was being displayed

It's probably helpful in this discussion to make a difference between two definitions of reasoning:

1. phenomenal reasoning, requiring consciousness and subjective experience

2. functional reasoning, transforming premises into conclusions using logic

I think you are attacking this using definition 1, whereas the article is obviously aiming at a different type of reasoning, and trying to formalize what is actually going on. It seems to be a genuine effort.

>1. phenomenal reasoning, requiring consciousness and subjective experience

I think it is incumbent upon anyone arguing that something does not posses any given property to provide a non-circular definition of what it is that they are declaring an absence of.

All of the descriptions of experiential reasoning are usually defined in terms of rephrasing of the claim "true understanding", "conscious", "aware", "knowing" all hinge on a synonymous aspect of the words that try and shift the responsibly of explanation to the next term used in a cyclic manner.

For the weaker sense of reasoning, there simply isn't any argument that it is not happening. A calculator can perform the weaker sense. The analysis of this aspect of LLMs is purely a question of how, not what.

It’s curious how they solve unsolved math problems without reasoning. Maybe I have a different definition of reasoning than you.

Guess what? SAT solvers have also solved unsolved math problems. Do you believe they are “reasoning”?

SAT solvers are programs designed such that their execution corresponds to the reasoning process of satisfying some given constraints. But they do not contain the reasoning process, rather they embody it.

LLMs are different in that they operate on semantic features of program state. Embedding vectors assign semantic features to syntactical structures of the vector space. Operations on these syntactical structures allow the LLM to engage with semantic features of program state directly. Here the reasoning process is contained within as an object of manipulation. An LLM sensitive to the semantic features of the input sequence and that examines the logically permissible moves to derive a new sequence closer to the intended sequence (some statement to prove) just is engaging in reasoning.

The question of whether a SAT solver can reason is about as interesting as the question of whether a submarine can swim. (EWD867, EWD898)

I think you are missing the point of that statement

It is a claim that swimming is a word that defines a context. It is an explicit statement that the question of whether a submarine can swim has nothing to do with the capability of the submarine.

If you are asking which pigeon hole we are putting something into, the answer is "The one we put it into". This is what make the question uninteresting.

If you are asking what is it about this pigeon hole that people value and does that align with the criteria that people use to decide categorisation. That very much is an interesting and complicated question.

The statement takes meaning-as-use as a given, sure, but I think the point of the statement is that people are arguing over an uninteresting question / taking meaningless positions about a meaningless issue, rather than "hey, words are moves in a language game!". I referenced two EWDs, which provide the original statements in context (though I can't find the widely-quoted wording anywhere: I thought I remembered it being in EWD1035, but apparently not). If you think my understanding of what Dijkstra meant was wrong, could you explain further, please?

I take your point that this seems to be the implication of what Dijkstra was getting at. But the term itself is not evocative for that reason. It resonates because people clearly don't think it is a meaningful question as to whether or not submarines swim because the term swim itself implies the categorisation I mentioned in my post above.

I do not know whether Dijkstra understood this distinction and was using it to disingenuously imply that the limitation was on the target and not the categorisation. He may have just felt it resonate with himself and failed to explore why.

Dijkstra immediately before using the term throws shade on serious thinkers engaging in a topic seriously. He personally seemed to want to dismiss the issue out of hand. As such I don't think there is any real value in his opinion on the matter. A recognition of how people did take it seriously and a considered rebuttal would be worthwhile. Declaring it uninteresting and failing to engage in the arguments is simply opting out of the debate.

Jury is still out on this one.

This needs to be routine to be given asevidence…

…Unless you know exactly how the llm was trained and then how it was applied

Here is Yuji Tachikawa from Japan (Mathematical Physics, String Theory, QFT) on recent progress in his own work using Fable 5 :

"I've been trying out Claude Fable recently, and last night, on a whim, I showed it my research notes about a collaborative project that's seen no progress in the past six months or so and asked for its thoughts. To my surprise, it made a non-trivial observation and essentially solved it."

"I was also surprised that it was using sympy to automatically write code and verify his own predictions."

"Fable probably seems like it properly understands string theory and has intuition too—that's my impression"

With that definition, computers don't play chess, they just move the pieces using some weights and backtracking.

Stochastic gradient descent can be likened to traveling down a billion-dimensional canyon. But inference? Hardly.

i love how anthropic puts out some bs like this every few weeks 'we saw some red bridge lights blinking in model weights when someone mentions sfo. Arent they just like us?"

Compression is the trick. Its even philosophed about if compression = intelligence.

The LLM has to compress everyy question/prompt into its system. It does so by creating rules and ways of processing data (this can lead to AGI, world models or an architecture of sub architectures like an LLM + something else). So if it should respond in a way that only reasoning people can achieve, it might be able to learn a representation of what we call reasoning.

It read enough text in itself to even know about the concept of reasoning and how you would do that.

Even if this is only stochastic, it shouldn't be so devalued as your comment comes across.

Who says that we are doing anything more magic?

So, the best of compression is a procedural system, capable to generate from minimal input a whole world- basically just using abstract rules and metrics for reasonableness? So- AGI is in the principles behind universe generators like NoMansSky? Is there a ultimate game- that generates this from physics up?

Yes + the capability to use this 'world' to simulate outcomes. Thats exactly what LeCun and others are trying to do with world models.

NoMansSky generator misses the complexity of encoding more than just geometry and basic quests but thats one reason why google and others show you machine learning based 'game engines' which allow you to enter these worlds. They are not doing this to replace real game engines, the do this for world research.

The ultimate game is us. Our physics constants :D

Compression, but also generating , using experience and previous abstractions and evaluating new state from entropy by comparing it to previous discoveries

My toaster doesn't reason, and neither do the current clankers.

How'd your toaster do at IMO last year?

I hear he got burned pretty badly.

They don't reason.

What would change your mind?

Do they ?

The article answers this question, at least to the extent it can be answered, at this time.

We see some signs of reasoning, but also we understand little about how they work.

Do we see actual signs of reasoning or is it anthropomorphism? We have an innate tendency to do so as humans.

> Do we see signs of reasoning or is it anthropomorphism?

This is the part that so many folks just don't seem to understand (probably because it's been labeled as "thinking" or "reasoning" mode, and people assume that words have meaning). It's not reasoning or thought. It's spewing tokens pretending to "think", but it's actually just generating extra "context" to help the final answer be more coherent. The model isn't doing anything it doesn't already do. It's just doing more of it to improve the quality of the final answer displayed to the user.

You're describing a process by which a 'thinking' entity uses cognition to refine a solution to a stated problem. That's a lot of words so usually we shorten this to 'reasoning'.

Do LLMs 'think'? I 'think' they do in a way. I don't really know how I think myself but I know I do and therefore I am (thanks, Descartes). I have a somewhat better grasp of the way LLMs 'think'. They do so sequentially, building a chain of descriptors which best fit the problem and the preceding descriptors. I suspect I do something not entirely dissimilar- i.e. I imagine 'worlds' which are like the current one changed in some way so they the problem I'm working on is reduced, then refine those until it is resolved - but in a massively parallel way.

[flagged]

I've read a bit through your comment history which gives me the impression of a mostly rational individual with whom I agree on some things while disagreeing on others. You don't seem to be a raving anti-LLM crusader nor come across as a starry-eyed LLM fanboi. I therefore conclude that these last rants were, to speak with Ebenezer, the result of an undigested bit of beef, a blot of mustard, a crumb of cheese, a fragment of an underdone potato rather than a farewell.

Assuming this to be the case my real question is: what makes you so sure these things don't "think"? This question can only be answered if we first know what "thinking" actually entails. Sure, LLMs are mechanistic and deterministic, feed them the same quote and seed and they'll produce the same output, token for token. If what they do is "thinking" - albeit mechanistically - then it seems to give lie to the concept of free will since the output for a given input only depends on the seed value. Surely humans don't 'think' like that? Well... who knows? The 'neural network' in human brains is far more complex than the ones used to run LLMs while LLMs can have access to more 'factoids' than the average human. What comprises 'thinking' as we do it? What would happen if you give, say, the neural circuitry in a rat brain access to enough storage to contain the training data used in current LLMs? Can a machine ever be made to 'think' or is that something which will always be limited to living organisms? If the answer is 'yes' we're back at the definitional question of what 'thinking' entails, if it is 'no' we're entering more in the realm of metaphysics and religion.

I don't know what 'thinking' entails, I just know I do it. I therefore can not definitely state whether LLMs 'think' or 'reason' but I can apply reason to what I observe and know about how these things work. Those observations and that knowledge lead me to conclude that, absent some metaphysical or religious veto these models can be made to 'think' and might already be doing so.

You phrased it better (and with a lot more patience) than I ever could muster.

So many people I meet are so deeply convinced LLMs absolutely cannot physically think, because they define "thinking" as "that thing you do with your human brain where neurons are involved", and they define "LLM thinking" as "that thing ChatGPT does where it says it's thinking but it's actually just detached inference".

The underlying assumption is usually two-fold:

1. That simulated thinking is not thinking.

2. That "LLM thinking" is always only defined as Chain of Thought.

Well, 1 is a pretty useless stance to have, because it removes space for any useful definition of what thinking is. And 2 is simply false, as presented by Anthropic here: https://www.anthropic.com/research/global-workspace

To be 100% honest, I really don't want to have this (or any other) conversation with you (or anyone else) right now (or possibly ever; I'm just sick and tired of people and their hateful bullshit), but purely out of respect for your fair and balanced reply I'll try to respond here in a civil and honest way to the best of my ability. This really is the last response I intend to make to anything anyone says on the topic, so I'll try to be thorough. If you want to respond to anything I say here, feel free, but don't expect a reply after this one (I may possibly, but probably not) . That having been said, here goes...

> ... "impression of a mostly rational individual with whom I agree on some things while disagreeing on others."

I try, really I do. It's gotten really hard these days. You're welcome to agree or disagree; Totally normal and expected. I just get tired of getting shut-down on every little thing I say by so many people who have sometimes less than zero experience in the topic they claim absolute certainty about, no matter if I can trot out a parade of facts proving my points. This inevitably leads to stress that is no longer as easy to just "brush off" as it used to be. Sorry for that.

> "You don't seem to be a raving anti-LLM crusader nor come across as a starry-eyed LLM fanboi."

You're right. I'm neither. I am actually quite impressed and amazed with what LLMs are capable of (especially in the hands of skilled and knowledgable users) but I also understand fully that there are tradeoffs involved and responsibilities involved in the usage of such tools. I do believe they (and other "AI" related technologies) have huge potential for both good and bad (largely dependent upon the user and their intent) and like any new tool, I genuinely do hope this one finds more of the good use than the bad, but more and more I'm feeling like it's just gonna get weaponized against society at large. Sad, but nothing I can say or do will change it. I'm fully convinced of that at this point.

> "what makes you so sure these things don't "think"?" ... <more stuff said here> ... "If the answer is 'yes' we're back at the definitional question of what 'thinking' entails, if it is 'no' we're entering more in the realm of metaphysics and religion."

So, in my mind, "thinking" is a much more "active" process than the "calculation" done by a machine just mechanistically working through a bunch of math. Does a desktop calculator "think"? Does a mechanical device like an Abacus or anything else that can "do math" without electronics? Calculation isn't necessarily "thinking", even though thinking can (and often does) result in calculation.

Now, where I'm coming from with my assertion that LLMs don't actually think is due to a few factors. First off, I've been learning the mathematics involved in how these things work for a very long time (decades now actually; as "neural network" technology and ideas is truly not a new thing), and while it's really amazing stuff, it's not magic. It's just math. Really fancy and complex math, but still just math. As soon as the math stops being done, the "thinking" stops. Does a brain ever stop thinking? I get the impression that until death it's kinda always active, even when you sleep. Not so with an LLM. You give it input, a buncha fancy math gets done by a really powerful "calculator" (computer), it responds with output, then it stops until it gets another "trigger" to start calculating some more.

There's some very real flaws in seeing that process as thinking however, even if you're only talking about that time during which the calculations are taking place. The problem I see there is that the LLM cannot "second guess" itself or worry about whether it might be incorrect about something. It just forges ahead with the calculations and gives the end result to the user, right or wrong, as it was designed to do. It has no "skin in the game" or reason to care (even if it had the ability to care) and it's got no real sense of "self" or the world or anything. It's just doing some really amazing math that results in an illusion of a thought process.

That having been said, I'm firmly convinced that even as these things stand now, they can absolutely assist humans in their thought processes if used properly and judiciously with full understanding of their limitations and weaknesses taken into account. I just don't believe that "more of the same" will somehow magically become "sentient" someday without a huge advance in both the hardware and software technologies it's built upon (on the level of the "positronic brain" or some kinda hand-wavy "quantum technology" science fiction concept). Pretty darn certain that more massive "AI data centers" aren't gonna lead to a "magical thinking machine" with the current forms of "AI" we're working with.

> "Those observations and that knowledge lead me to conclude that, absent some metaphysical or religious veto these models can be made to 'think' and might already be doing so."

Now, this here I can actually agree with, other than the "might already be doing so" part. They're not (yet). I'm really quite sure of that, knowing what I know about how these things work. They really are fantastic at faking it these days though, as evidenced by how many people truly are buying into the AI company CEO hype about AGI/ASI. I think that LLMs can absolutely be one part of a machine that's capable of a simulation of "thought" that could really be good enough to qualify as some form of "the real thing" on some level, and that may even someday (soon even?) surpass the capabilities of humans in that regard. It'll require some different ways of doing things though, and some combinations of classic traditional computing with a wide range of related "AI" technologies including LLMs, neural nets, vision models, etc, etc, and it'll have to be put in some sort of active state of operation where it's capable of doing the "thinking" and "learning" process continuously the way an actual brain does. I think it'll also help to give it access to a continuous input stream similar to how a brain has access to near constant input as well.

Anyone that wants to really know how this stuff works "under the hood" is welcome to ask an LLM about it. Many of 'em are actually quite good at explaining themselves, starting from "first principles" if you ask 'em to "keep it simple" all the way down through the deep mathematics involved. I encourage folks to have that discussion with several of their favorite LLMs if for no other reason than more knowledge about the topic is a good thing. Just be aware that they can at times say things that are actively incorrect and they will often say such things with great certainty (and sometimes even try to argue with you about it if you call them out on it). Always check your own (and the LLM's) knowledge against known verifiable provable facts. This stuff is all heavily documented and readily available "out there" on the Web with not too terribly much heavy searching required.

To summarize; I don't think it's impossible to create a "thinking machine" using these technologies. I just don't believe we're even remotely nearly as close to it as the AI mega-corporations would have us all believe. I might be wrong about everything I've said here, or I could be 100% correct. Dunno; No longer care either way really. I've said my piece and I'm done now. Bring on our AI overlords, for better or worse. I can't stop it either way.

I don't know why you're saying you're "tired of people's hateful bullshit". I originally responded that your point was off, and you're the one writing these comments: https://news.ycombinator.com/item?id=48886350

Take a breather, my guy. Stop saying in every comment that "you're done" and "this is your last response" and actually go touch a bit of grass. You do not owe anyone on this website a response, and the only thing your comments are for (when they're good and not hostile) is contributing to a discussion. Nobody here cares whether you respond or not.

Anyway, not to detract - this last comment was good. I just think maybe you're putting intent in people's mouths where there is none. For example, I certainly don't believe the bullshit Altman and Amodei are putting out. I just disagree the LLMs don't think.

And yes I also understand the math behind them. But it being math doesn't mean there cannot be emergent behaviour, just like there is emergent behaviour after the layers upon layers of biology in humans, resulting in thinking. "It's not magic, it's just [biology]" applies to us as well.

"But brains are the most complex machines in the universe" some might say -> Right, but who's to say the thinking we do requires machines as complex as our brains are? Our brains move muscles, help us breathe and manage millions of other invisible innate processes, many things LLMs do not need to do. And LLMs have a very, very deep access to and understanding of language, which is argued to be a significant contributor to how humans think (based on studies on nonverbal humans).

> As soon as the math stops being done, the "thinking" stops. Does a brain ever stop thinking?

Right, so I understand your point in this and there's something to that, but it enters in how thinking is defined in both ways.

First, the calculation not running is at best equivalent to time stopping. It's not like the LLM actually sits there waiting for input. As you said: we continuously receive input, and we do so because our biology is built this way and we are damn energy efficient, so we can afford to continuously and asynchronously process that input. But this is not really related to the process of thinking itself.

So it comes down to what happens during the calculation. And Anthropic's published research on the J-space is pretty damning evidence, IMO, that thinking does happen during the calculation.

Yeah, you can just fuck right off, too. My reply wasn't for you. It was for the person who wasn't actively trying to rub salt in the wounds and be a worthless little shit.

> I don't know why you're saying you're "tired of people's hateful bullshit". I originally responded that your point was off, and you're the one writing these comments: https://news.ycombinator.com/item?id=48886350

You really can't figure it out? Did I not make myself pretty clear in the comment I was responding to there that it's decades of shitty people like him (and apparently you) that have pushed me to this point? Funny how you point to my comment without noticing or acknowledging at all the fact that it was a response to another hostile comment that was all about just tryin' to be hateful and ignorant and push me even further into hatred.

> Take a breather, my guy.

I'm not "your guy". I actively hate you.

> ... and actually go touch a bit of grass.

And this sorta shit right here is why I hate you and everyone like you. You know what I did yesterday? I turned my computer off and went and "touched grass" (hung out in the park away from technology and people) and just chilled out. You make it sound like I don't love nature or some stupid shit. You don't know me, so just keep your unhelpful advice to yourself.

> I just think maybe you're putting intent in people's mouths where there is none.

So it wasn't your intent to make snide little insinuations and remarks in an attempt to purposely try to rub me the wrong way so as to piss me off even more after I'd finally managed to calm myself down? Yeah, sure. I believe that... It wasn't me who put those remarks in your mouth. You chose to try to rub salt in the wound. Congratulations. You're human garbage. Good for you.

Talk to a therapist.

> Talk to a therapist.

Why? It might help me but it won't fix you and you're the problem here. Your whole intent is to aggravate me into hating you even more than I already do. Well, guess what. You're wasting your time at this point. I couldn't possibly hate you more than literally wishing you didn't exist at all. You've already succeeded at your goal, so you can just shut the fuck up now and go away. Congratulations. You win. Be happy and be gone.

[flagged]

Honestly, people need to get over this debate. It's pretty irrelevant in a lot of cases. When people ask "what is the model thinking?", they're really asking "what caused the model to produce this response (as opposed to a bunch of other plausible ones)?"

Whether it's thinking or word prediction or whatever you want to call it, people are trying to understand the causal chain.

It's not just a nominalistic debate though, as the people who are vocal against the idea that LLMs might "understand" or "think" also claim that because of this, they are fundamentally limited in what they can achieve, in contrast to human beings. Therefore any possibility of actual intelligence (or even superintelligence) is, according to them, just a fantasy.

Angry diatribes about whether submarines swim or not.

Yes, we do see signs of actual reasoning, see the papers linked in the article. (There are many others too.)

Yes, we have a tendency to anthropomorphize, but (most) researchers are aware of this.

The papers linked in the article discuss the mechanical operations that simulate reasoning. Intelligence is data efficiency and I don't see a strong argument that reasoning can exist if it requires a world's worth of data.

That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.

> a thing is not its simulation.

"The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circuits glowed red and gold as she picked simulated daisies, and hummed a simulated song."

(Stanislaw Lem, Cyberiad)

"In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.

"Suarez Miranda,Viajes de varones prudentes, Libro IV,Cap. XLV, Lerida, 1658"

- On Exactitude in Science by Jorge Luis Borges

Yes, there is an LLM feature that we have anthropomorphized as "reasoning" or "thinking", where an LLM has a scratch space where it can dump tokens that help to improve the final output.

> that help to improve the final output

Do they actually help? Are you sure?

Of course they do, how else do you think they manage to implement new features in large codebases, or to prove new theorems? But you don't even have to assume they do because of the results- you can read their chain of thought.

The Eliza effect.

It's indeed so powerful that even my compiler and my unit tests fell victim of this delusion.

Reply on news.ycombinator.com