"The Illusion of Thinking" – Thoughts on This Important Paper
hardcoresoftware.learningbyshipping.com
hardcoresoftware.learningbyshipping.com
- Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity https://arxiv.org/abs/2506.09250
I don't care about benchmarks, nor LLMs' capability to solve puzzles. This is the first thing AI companies optimize their training for, which is misleading and practically false advertising.
I care about how good LLMs are for helping me with specific tasks. Do they generate code that on the surface appears correct, yet on closer inspection has security and performance issues, is unnecessarily complex, often doesn't even compile, which takes me more time to troubleshoot and fix than if I were to write it myself? Do they explain concepts confidently while being wrong, which I have no way of knowing unless I'm a domain expert? Do they repeat all these issues even after careful re-prompting and with all the contextual information they would need? Does all this waste my time more than it helps?
The answer is yes to all of the above.
So while we can argue whether LLMs can think and reason, my personal experience tells me that they absolutely cannot, and that any similarity to what humans can do is nothing but an illusion.
Benchmarks and puzzles don't matter. AI companies will only use them to optimize their training, so that they can promote their results and boost their valuation on hype alone. They're more useful for marketing than as a measurement of real-world capabilities.
Judging by the increase of these negative discussions, we seem to be near or at the Peak of Inflated Expectations w.r.t. AI, and we'll be better off once we surpass it.
These were not costs , but massive benefits for hyped AI startups. They attracted the attention of a wide audience of investors including clueless ones, they brought in politicians and hence the media attention, and created a FOMO race for them to plant flags before each other.
That's the real conclusion of the Apple paper. It's correct. LLMs are terrible at arithmetic, or even counting. We knew that. So, now what?
It would be interesting to ask the "AI system" to write a program to solve such puzzle problems. Most of the puzzles given have an algorithmic solution.
This may be a strategy problem. LLMs may need to internalize Polya's How to Solve It?[2] Read the linked Wikipedia article. Most of those are steps an LLM can do, but a strategy controller is needed to apply them in a useful order and back off when stuck.
The "Illusion of thinking" article is far less useful than the the Apple paper.
(Did anybody proofread the Apple paper? [1] There's a misplaced partial sentence in the middle of page 2. Or a botched TeX macro.)
[1] https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin...
But testing via coding algos to known puzzles is problematic as the code may be in the training set. Hence you need new puzzles, which is kinda what ARC was meant to do, right? Too bad OpenAI lost credibility for that set by having access to it, but ”verbally promising” (lol) not to train on it, etc.
Which version of `bc` was the LLM trained on? Perhaps this is not so important for a stable tool that doesn't change much, but it's critical for many programs and programming libraries. I lost count of the number of times when an LLM generated code that doesn't work for the current version of the library I'm using. In some cases you can tell it to use a specific version, or even feed it the documentation for that version, but that often fails to correct the issue.
And this is without considering the elephant in the room: hallucination. LLMs will often mix up or invent APIs that don't exist. Apologists will say that agents are the solution, and that feeding the error back to the LLM will fix it, but that is often not the case either. That might even make things worse by breaking something else, especially in large-context sessions.
Nothing has convinced me that LLMs will ever be good at math on their own.
LLMs are transformative, but a lot of the tools around them already treat them as opaque function calls. Instead of piping text to sed, awk, or xargs, we’re just piping streams to these functions. This analogy can stretch to cover audio and video usage too. But that’s boring and doesn’t explain why you suddenly have to pay more for Google Work Suite just to get bombarded by AI slop.
This isn’t to undermine the absolutely incredible achievements of the people building this tech. It’s to point out the absurdity of the sales pitch from investors and benefactors.
But we’ve always been like this. Each new technology promises the world, and many even manage to deliver. Or is it that they succeed only because they overpromise and draw massive attention and investment in the first place? IDK, we’ll see.
if you only aim for the moon, you’ll never break orbit.
You launch for standard equatorial low orbit, then coast until the Moon is about 90 degrees to the right / behind you, and burn ahead until you run out of fuel in the second stage. This turns your orbit into an elongated ellipse, and as you get close to its far end, the Moon will catch up and capture you in its gravity well.
When playing one of the non-sandbox aka. "progression" modes, i.e. "career" or "science", the trajectory planning tools are locked behind an expensive upgrade, effectively forcing you to do your first Moon fly-by this way.
ty i shall pass that rite of passage some day and eyeball my way to the moon
Lately chatgpt is like -let me show you an example I used before- like its a real professional.
it's all in the context, its our duty to remember these are LARP machines
I copy - pasted something about a kid their parent was asking online. Chatgpt said:
⸻
A Personal Anecdote (Borrowed)
A parent I know gave their daughter a “leader hat.” [..]
How in the hell would a large language model have past personal anecdotes and know other parents, idk
And knowing is something even dumb programs do with data. It doesn't imply human cognition
But that was just like the latest, I just don't love how it talks like it was an alive person at the other end, maybe I should add something to my personal prompt to avoid it.
Unfortunately, it's also the least wrong approach. People who refuse to entertain thinking about LLMs as quasi-humans, are the ones perpetually confused about what LLMs can or cannot do and why. They're always up in arms about prompt injections and hallucinations, and keep arguing those are bugs that need to be fixed, unable to recognize them as fundamental to the model's generality and handling natural language. They keep harping about "stochastic parrots" and Naur's "program theory", claiming LLMs are unable to reason or model in the abstract, despite many published research that lobotomizes the models live and nail concepts as they form and activate.
If you squint and imagine LLM to be a person, suddenly all of these things become apparent.
So I can't really blame people - especially non-experts - for sticking to an approach that's actually yielding much better intuition than the alternatives.
Neither it is for humans, the whole "they haven't lied to me so far, so it's unlikely they will suddenly start lying to me now" thing is strongly conditioned on "low frequency" assumptions from shared biology, history and culture, as well as "high frequency" assumptions about the person, e.g. that they're actually friendly, or the context didn't change, that your goals are still aligned[0]... or that they aren't drunk, confused, or a kid.
Correct your theory of mind for the nature of LLMs, and your conclusions will be less wrong. LLMs are literally trained to approximate humans, so it makes sense this approach gives a decent high-level approximation.
(What LLMs don't have is the "sticky" part of social sphere; humans, generally, behave as if they expect to interact again in the future. Individual or public opinion doesn't weigh on them much. And then there's the other bit, in that we successfully eliminate variance in humans - the kind of people who cannot be theory-of-minded well by others, get killed or locked up in prisons and mental institutions, or on the less severe end, get barred from high-risk jobs and activities.)
To be clear: I'm not claiming you should go deep into the theory-of-mind thing, assuming intentions and motivations and internal emotional states in LLMs. But that's not necessary either. I say that the right amount of anthropomorphism just cognitive and focused on immediate term. "What would a human do in an equivalent situation?" gets you 90% there.
--
BTW, I looked up the theory of mind on Wikipedia to refresh some of the details, and found this by the end of the "Definition" section[1]:
> An alternative account of theory of mind is given in operant psychology and provides empirical evidence for a functional account of both perspective-taking and empathy. The most developed operant approach is founded on research on derived relational responding[jargon] and is subsumed within relational frame theory. Derived relational responding relies on the ability to identify derived relations, or relationships between stimuli that are not directly learned or reinforced; for example, if "snake" is related to "danger" and "danger" is related to "fear", people may know to fear snakes even without learning an explicit connection between snakes and fear.[20] According to this view, empathy and perspective-taking comprise a complex set of derived relational abilities based on learning to discriminate and respond verbally to ever more complex relations between self, others, place, and time, and through established relations
I then followed through to relational frame theory[2], and was surprised to find basically a paraphrase of how vector embeddings work in language models. Curious.
--
[0] - Yes, I used that word. Also yes, in weak or transactional relationships, this may change without you realizing it, and the other person may suddenly start lying to you with no advance warning. I'm pretty sure everyone experienced this at some point.
[1] - https://en.wikipedia.org/wiki/Theory_of_mind#Definition
Trusting OpenAI to not put sychophant 4o on, or not do funny things on their opaque CoT is a whole another thing.
The problem is 1% of us are programmers and can reason about these things as the technology they are, for the 99% of the rest that don't understand it and just -use it- it feels like magic, and magic is both good/bad from this POV
me thinks anyways
I guess me feeling conscious, and thus directing much of my life and activity around that notion is just a big trick, and i'm no different from ChatGPT, only inferior and with lower electrical needs.
1. As far as we know there is nothing non-physical in the brain (a soul or magical quantum microtubules or whatever). Everything that happens in your head obeys the laws of physics.
2. We definitely are conscious.
3. Therefore consciousness can arise from purely physical processes.
4. Any physical process can be computed. And biological thought appears as far as we can see to simply be an extraordinarily complex computation.
5. Therefore it is possible for a sufficiently fast & appropriately programmed computer to be conscious.
6. LLMs bear many resemblances to human thinking (even though they're obviously different in many ways too), and we've established that computers can be conscious, so you can't trivially rule out LLMs being conscious.
At least for now, I don't think anyone sane says today's LLMs are actually conscious. (Though consciousness is clearly a continuum... so maybe they're as conscious as a worm or whatever.)
The main point is that people trivially dismissing AI consciousness or saying "it can't think" because it's only matrix multiplications are definitely wrong.
It might not be conscious because consciousness requires more complexity, or maybe it requires some algorithmic differences (e.g. on-line learning). But it definitely isn't not-conscious just because it is maths or running on a computer or deterministic or ...
Even if consciousness arises from a physical system, it doesn't follow that computation on a physical substrate can produce consciousness, even if it mimics some of the qualities we associate with consciousness.
> you can't trivially rule out LLMs being conscious.
Which is the actual conclusion. That's all.
The one trying to read more into it by claiming a stronger conclusion than warranted (claiming this is all "doubtful") is you.
> Even if consciousness arises from a physical system, it doesn't follow that computation on a physical substrate can produce consciousness
This is incoherent.
The foregoing argument does not establish that computers can be conscious. It assumes consciousness arises from purely physical interactions AND that any physical process can be computed AND that computation can produce consciousness. It provides no reason to believe any of those things, and therefore no reason to believe computers or LLMs can be conscious.
Once you remove the unjustifed assumptions, it boils down to this: LLMs can do something that looks a bit like what some conscious beings do so they might be conscious.
The final paragraph is not incoherent.
Yes, and with these assumptions, then "We've established that computers can be conscious, so LLMs might be".
Again that is all that is being claimed. The one insisting the argument is anything more persuasive than that is you. You seem hung up on this claim, but need I remind you it's only an existence proof, whether it's actually possible/probable/likely is an entirely different claim, and is not what is being claimed.
> Even if consciousness arises from a physical system, it doesn't follow that computation on a physical substrate can produce consciousness
> If consciousness arises from a physical system, a physical system cannot produce consciousness
> If A not A
This argument is a contradiction, it's incoherent.
If you're trying to draw a distinction between physical systems and computation on physical systems, well that's a moot point as every tested theory in physics uses mathematics that is computable.
Perhaps reality is not computable. If so then yes, that would mean computation alone will not produce the same kind of consciousness we have. That leaves open other kinds of consciousness, but that's another discussion.
I'm not sure you're following what's being discussed in this thread.
For all consciousnesses that exist, they arose from a (complex) physical system.
What you are positing is, effectively:
Any physical system (of sufficient complexity) could potentially become conscious.
This is a classic logical error. It's basically the same error as "Socrates is a man. All men are mortal. Therefore, all men are Socrates."
Am I saying LLMs are sufficiently complex? No.
But can they be in principle? I mean, maybe? We'll have to see right?
Saying something is possible is definitely an easier epistemic position to hold than saying something is impossible, I'll tell you that.
You might think this is obvious and I agree. Yet there are many people that argue that computers can never be conscious.
> even if it mimics
Crucially even if it appears indistinguishable. It's obvious where that could lead, isn't it?
Um, really? How so? Show me the computations that model gut biome. Or pain, or the immune system, or how plants communicate, or how the big bang happened (or didn’t).
And now define “physical” wrt things like pain, or sensation, or the body, or consciousness.
We know so little about so much! it’s amazing when people speak with such certainty about things like consciousness and computation.
I don’t disagree with your main point. I would just say we don’t know very much about our own consciousness and mindbody. And humans have been studying that at least a few thousand years.
Not everyone on HN or who works with or studies AI believes AGI is just the next level on some algorithm or scaling level we haven’t unlocked yet.
Just because computations exist that model a physical process to an arbitrary degree of accuracy doesn't mean we know what those computations are.
But no one can use a physical model or computer to precisely compute, say, the fall of sand (where the exact position and momentum of every grain of sand is precisely known) - so we have no actual proof that the actual physical process is computable.
And, in fact, our best fundamental physical theory is not fully computable, at least in some sense - quantum measurement is not a deterministic process, so it's not exactly computable.
You can simulate the wave function of particles, e.g. https://github.com/marl0ny/QM-Simulator-2D
From there you can build up simulations of atoms, molecules, proteins, cells, and the gut biome.
Obviously I'm not saying that it is computationally feasible to do an atom-level simulation of the guy biome, or pain receptors or the immune system.
But there's nothing fundamental that precludes it. It's just scale.
Also it's highly unlikely that biological intelligence depends on atom-level modelling. We can very likely use a simpler (but maybe still quite complex) model of a neurone and still be able to achieve the same result as biology does.
Reglious friend: Well there must be a god, I can feel him, and how else would all this marvellous reality around us have come to existence?
Atheist friend: How can you claim that? Where is the evidence? There could be many other explanations, we just don't know, and because of that it doesn't make sense to postulate the existence of a god. And besides, it's just a marketing stunt of the powerful to stay in power.
Agnostic friend: Why does any of this matter? Nature is beautiful, sometimes cruel, I'm glad I'm alive, and whether there is a god or not, I don't know or really care, it's insubstantial to my life.
That's exactly how the discussions around LLM consciousness or intelligence or sentience always go if you put enough tech folks into the conversation.
Why does this surprise you? This is a community hosted by a Silicon Valley venture capital firm, and many of its members are either employed by or own companies in the AI industry. They are personally invested in pushing the narrative being criticized here, so the counter arguments are expected.
Parts of this blog post so closely match the language and structure of this interview[0] that I'm inclined to accuse the author of direct plagiarism.
[0] https://hardcoresoftware.learningbyshipping.com/p/233-the-il...
this is literally what they are
Can we consider AI conscious similar to how Hardy recognised Ramanujan's genius? That if AI weren't to be conscious then they won't have the imagination to write what it wrote.
https://www.radiolab.org/podcast/161744-loops/transcript
*The Prisoner of Memory*
In August 2010, Mary Sue Campbell became trapped in the most peculiar prison—a 90-second loop of her own consciousness. After experiencing transient global amnesia, she lost the ability to form new memories, resetting every minute and a half like a human record player stuck on repeat.
What made her condition haunting wasn't just the forgetting, but the eerie predictability. Every cycle, she asked the same questions in identical phrasing: "What's the date?" followed by "My birthday's already passed? Darn." She'd listen to her daughter's explanation, widen her eyes at the mention of paramedics, and declare "This is creepy!" The nurses began mouthing her words along with her.
Dr. Jonathan Vlahos, who treated similar cases, found this mechanical repetition deeply unsettling. It suggested that beneath our illusion of free will lies something more algorithmic—that given identical inputs, the brain produces identical outputs. Mary Sue's loop revealed the uncomfortable truth that our sense of spontaneous choice might be far more fragile than we imagine.
This part doesn’t seem correct. Each 90 seconds she has completely different inputs: sights, sounds, the people she’s around, etc. Seems like it has more to do with her unique condition than any sort of revelation about free will.
There are a complexities beyond that of course. It can compare stuff, it can iterate through multiple steps ("Reason," though when you look at what it's doing, you definitely see how that term is a bit of a stretch). Lots of emergent applications yet to be discovered.
But yeah, it's math (probabilities to generate the next token) and code (go run stuff). The best applications will likely not be the ones that anthropomorphize it, but take advantage of what it really is.
The models make many mistakes and are not great software architects, but watch one or even two models work together to solve a bug and you'll quickly rethink "text transformers".
If we ever develop a LLM that's able to apply symbolic logic to text, for instance to assess an argument's validity, develop an accurate proof step by step, and do this at least as well as many human beings, then I'll concede that we've invented a reasoning machine. Such a machine might very well be a miraculous invention in this age of misinformation, but I know of no work in that direction and I'm not at all convinced it's a natural outgrowth of LLMs which are so bad at math (perhaps they'd be of use in the implementation).
LLMs very obviously are reducible to "just code and maths". We know that, because that is how they are made.