LLMs and self-referentiality | Hacker News Reader

LLMs and self-referentiality

scottaaronson.blog

83 pointsmavdol0494 comments

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.

But I suppose it doesn’t harm its reasoning!

Any signal that allows the model to see what's wrong helps.

Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct.

Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.

> Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.

A global database of facts would make easier for AI to stay factual

Also ironically it would also make it easier to align AI to do things like consistently censor or distort some political facts

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

It’s really not like that. If I ask you to tell me about a geographical place I just made up, you can trivially and generally instantly recognize you don’t recognize it. A child can do this. You wouldn’t be able to hold a job or generally get through life without this level of self awareness.

So can LLMs. I asked one about South Wollopop and it suggested that it'd never heard of it but maybe I meant South Wollo in Ethiopia. And that's just a local model with modest hardware and no internet access. First attempt.

I mean, if our position is that a hyper advanced statistical model is going to ultimately struggle with the concept of something being unlikely to be true then the statisticians may as well give up in despair. There is no theoretical obstacle here.

Current frontier models are pretty good at telling you when they don't know something, or if you're asking about something nonexistent or nonsensical. Not always, but there has been massive improvement on this front lately.

If your last experience with frontier LLMs (read: not models that Google and ChatGPT are giving away for free, but models you have to pay for) was over a year ago, you may not realize that.

I had a great game as a kid where me and girls were playing for kisses on "words", i.e. "start the next word with the letter of the word I just said". So I invented a whole new vocabulary and then taught them that vocabulary. Apart from the fact that I got a lot of kisses from them they were so happy they learned something new they immediately went to share this with their (and my) parents. Obviously I got a slap but I just don't get how you can think "a kid can do this". You're either completely ignorant of different cultures (i.e. as an Eqstern European I still find it hard to remember Indian names, let alone tell if they are actually fucking with me) or intentionally simplifying the problem.

No, I’m referring to something else. Not the next word task. Being able to know whether you know something. Next time you encounter a child, ask them something they’re likely to be ignorant of, and see they’re able to say “I don’t know”.

But do you really know? Hence stuff like mansplaining where people vastly overestimate their knowledge compared to average Joe

From your comment, I noticed a sort of pattern that is often seen when discussing LLMs. That they are these amazing things that can run so fast they trip themselves in their attempts at achieving a task. So we resort to refining the models, creating guardrails, orchestrating harness, so as to alleviate the 'hallucination' problem.

In contrast to human intelligence, there is an underlying mechanism that propels intelligent behaviour. A person is no less intelligent just because they lose sight, sound or inner voice.

“trip themselves in their attempts at achieving a task”

I see this a lot in Claude Code. I assume it has to do with the training structure.

Example is “fallbacks”. Claude constantly sprinkles “fallbacks” in the code, even when I ask it not to. That is, write multiple candidate implementations into the same code with some kind of switch.

This is a problem because you only need one, and it would seem to have inflated the code for no reason. (The madness accelerates with code volume, so you must push back.) anyway, few people would do it this way.

But I thought, what could be the benefit?

If you’re being conditioned to pass evals one-shot with code that will be discarded and never read, it’s a great strategy. If you have more than one way to solve it, you can just put both. The behavior would easily be reinforced, if trained that way.

But in any case, again, a certain nature and certain conditioning.

I think we’ll learn to accept it as AGI but also that no intelligence is fully divorced from context, limits and conditioning.

Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition.

Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.

You should read his later work called I Am a Strange Loop instead of GEB, as this is the author’s preference.

I think of this book often in the LLM age.

Thanks. I actually read this book in high school. I remember being very impressed with it and bringing it to school.

I don't remember a single thing! (Which is unusual for me, I usually remember much of what I read.)

I shall have to read it again :)

Two comments on this, trying to take a "which hypothesis fits the evidence" approach.

First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what it's read about them. In the same way, "non-aware spicy autocomplete" would produce an LLM that spoke about its own experience, based only on the input it has of people speaking about their own experience.

That said / secondly, from the little I understand of LLM architecture, I believe there are a large number of self-referential mechanisms built in. For one, nearly all transformers have a "residual layer", with various neural networks essentially reading and modifying it. This effectively forms a loop. Additionally, the "thinking" mechanism allows it to read what it's written and generate more things, which is again a loop.

So, maybe people didn't think, "Hey, we should build some loops, maybe that will make it conscious". But if "strange loop" is what defines consciousness, there are lots of loops in there onto which such a strange loop could conceivably form.

> First, an LLM describing its own experience is not actually proof that it has any experience to be aware of,

Likewise, humans describing their own experience is not actually proof that they have any experience to be aware of

Indeed, but I actually kind of mis-spoke here. The question is less about having an experience to be aware of, but the ability to accurately reflect internal state.

Even humans need to learn how to read their own internal state (e.g., saying "I got mad" rather than "I felt ashamed because I wasn't living up to my picture of what a good person is, and covered up the shame with anger").

But if an LLM were to say, "I'm sad" or "I'm happy", does that actually correlate to anything? I'm OK with saying "The LLM was sad", if there is an internal state that leads to observable changes in behavior correlating with the kinds of changes in behavior humans have when they're sad. The question is, if the LLM says "I'm sad", is that because it has that internal state (self-reflection)? Or is it because that's the kind of thing a human would say in that context?

I think both are possible. I also think that between internal probes and behavioral testing, it should be possible to determine which one is closer to the truth. I'm just pointing out that "LLMs talk about their internal state" isn't proof that LLMs have self-referentiality, without additional evidence that the talk is actually related to their internal state.

> But if an LLM were to say, "I'm sad" or "I'm happy", does that actually correlate to anything? I'm OK with saying "The LLM was sad", if there is an internal state that leads to observable changes in behavior correlating with the kinds of changes in behavior humans have when they're sad. The question is, if the LLM says "I'm sad", is that because it has that internal state (self-reflection)? Or is it because that's the kind of thing a human would say in that context?

If they are not trained specifically to find those internal state correlates when introspecting, I would find it quite shocking to see that introspection is an emergent behavior of LLMs

‘But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence?’ — has anyone ever formulated this actual idea or anything logically equivalent? This looks like a mighty straw man.

I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.

They don't only reference a frozen training reality, they also reference the live rollout. Otherwise they wouldn't work at all.

They do continuously update from experience, but that only lasts for as long as the current rollout lasts.

Was Hofstadter ever arguing that intelligence requires self-referentiality?

I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.

I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.

Yes -- I can't remember the exact words, but there's a striking bit in GEB where he directly addresses the question of "will a machine ever create art?" His answer is yes, but only after it has really lived life, experienced heartbreak, and so on. The machine would have to have a self, which he argued arose from self-referentiality and "strange loops".

Oh, I guess there's a question of whether you can separate intelligence from consciousness. His writing was mostly done in the long period where things that are very easy for humans were still intractably difficult for machines -- identifying objects in pictures, following simple written instructions, etc. So I think it's clear that he saw "intelligence" as being able to solve those simple-yet-intractable common-sense problems; the kind of thing LLMs now excel at.

> His answer is yes, but only after it has really lived life, experienced heartbreak, and so on.

The thing about this is that we're always encouraging people to read, because it gives them access to experiences and exposure to ideas far beyond what they could just speaking to the people around them. But there's no human alive who has read as widely or esoterically as the current crop of frontier models.

A few quotes towards the end of GEB:

> I see no barrier imposed by Gödel’s Theorem to the implementation on computers (or their successors) of types of symbol manipulation that achieve roughly the same results as brains do.

And a distinction between intelligence and consciousness:

> It is entirely another question to try and duplicate in a program some particular human’s mind—but to produce an intelligent program at all is a more limited goal.

And then:

> Gödel’s Theorem doesn’t ban our reproducing our own level of intelligence via programs

That is pretty hard to square with "GEB argued that AI couldn’t achieve intelligence without first mastering Gödelian self-reference."

A bit more introspective:

> I think that the process of coming to understand Gödel’s proof, with its construction involving arbitrary codes, complex isomorphisms, high and low levels of interpretation, and the capacity for self-mirroring, may inject some rich undercurrents and flavors into one’s set of images about symbols and symbol processing, which may deepen one’s intuition for the relationship between mental structures on different levels.

Perhaps "true" self-referentiality was not needed. But it seems "dynamic feedback" is essential, and it seems to be adjacent to self-referentiality.

If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.

Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.

Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.

I've used concepts from GEB a lot in my thinking about how LLMs.

On self-referentiality and the MU Puzzle: https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...

On Gödelian limits of prompt-safe AI : https://matthodges.com/posts/2025-08-26-music-to-break-model...

On the blurry line between pattern matching and reasoning: https://matthodges.com/posts/2026-08-19-bongard-problems/

I'm glad that the author of this post ends with:

> What’s left? Consciousness

because when I've read (and re-read) GEB, I find the book to be much deeper than just a threshold for conversational intelligence.

What the author refers to is the ability to iterate and update the internal memory. A classic transformer based LLM can only produce the next token and never go back and update old tokens or delete them. The best thing it can do is produce thinking tokens to serialize the internal state of the final layer so that it can pass it back into the first layer.

Just like the transformer was an advancement over LSTMs by making it possible to have perfect recall (reading every input), the only way to improve over the transformer is to build a deep equilibrium version, where the DEQ transformer is capable of updating its own memory (writing every output).

Such a machine would be considered a linear bounded automaton (a turing machine without unlimited tape) and therefore even the human brain could not have an architectural edge over it in terms of intelligence. The human brain could only have an edge in terms of energy efficiency.

> It claimed (incorrectly, in my view and in most experts’) that AI could never work because there was something about Gödel’s Theorem and self-reference that no computer program could ever capture

The fact he uses an appeal to “the experts” here for something that is fundamentally a philosophical or metaphysical claim shows he doesn’t actually understand the point.

If LLMs were equivalent to humans then we wouldn’t use them. They would be doing their own thing according to their own will.

I still need to tell the LLM what to do and to direct it according to my will in order to create something that is useful. And I say this as someone spending $600/month on codex and Claude max subscriptions. I’m managing these things all day

This one thinks LLMs should be trained and RLHF'd into illeists, as an experiment.

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious?

We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really define a marker or a boundary beyond which I would start calling them "conscious."

It's not in big-money's interest to call them conscious because that would mean they get rights too.

Consciousness requires qualia. My 0,02€.

We don't understand what those are or how they come about, so how does that help us objectively reason about whether a particular black box does or does not have them, especially when it behaves as though it does, beyond arbitrary I-know-it-when-I-see-it "things I don't like don't have the property" style declarations?

That's the definition that matters. And as it's impossible to describe in words what it's like to experience qualia, and LLMs are trained on words, they cannot know what it's like. Even if images and sounds are included in their input training data, those are just arrays or sequences of numbers, not the experience of seeing and hearing.

Their training data isn't qualia itself, but neither is your training data.

Qualia are something you get on the inside in response to inputs. I don't see why LLMs can't have them. I'm not saying they do, but I don't think your argument proves that they don't.

My training data on qualia is actually seeing, hearing, and otherwise sensing things, not just reading about them.

Ah yes, because you used the magical pixie dust that makes experiences "real" therefore you're actually seeing and hearing while a multimodal LLM doesn't really experience what it sees and hears "for real". We can be certain of this because, despite not being able to quantify or even rigorously define the phenomenon we intentionally deprive the LLMs of access to said magical pixie dust.

I agree. To further ask, why is your training data objectively real and qualifies you for consciousness?

Is a deaf or blind person not conscious because they don't share your training data? If a deaf person reads about sounds, their experience of them is moot?

If an alien with a greater array of senses then us exists, are we therefore not conscious?

People aren't just consciousness, they're conscious of things. Of course a deaf or blind person is conscious, even if they've been deaf or blind since birth. But they aren't conscious of hearing or seeing things, and if you ask them they'll tell you that.

I'm not conscious of magnetic fields, or ultraviolet light, but other animals are, and no amount of study will overcome that deficit.

Okay, but there are multi-modal models. So do those experience qualia of sound and images? Do the text based ones experience the qualia of reading, or of finally comprehending a difficult concept after struggling with it?

For that matter, what exactly is the difference between "seeing" that a car is red versus an LLM reading an array of pixel data? Your eyes "just" translate photons into analog signals after all.

I've already answered this. There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of, and if you don't believe me you can ask them. Or by looking at a speech spectrogram (which, with practice, are more easily readable).

Or you can ask yourself what ultraviolet looks like, or what it's like to sense a magnetic field. Lots of animals can do these, and would know what it's like, but people can't.

No, you have not answered. I asked you what you believe the difference is between the two phenomena in concrete terms. What you've done is beg the question.

> There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of

On what basis could you possibly make this claim? What does it mean to hear? You're taking in information and decoding it, right? So what magical pixie dust could your eardrum possibly be applying that specially privileges that particular mode of input?

How is an LLM "reading" pixel data fundamentally different from your eyes transforming it into an analog signal?

Please be specific about the mechanism.

> On what basis could you possibly make this claim?

People with absent qualia have been asked, e.g. Knut Nordby: https://www.consc.net/misc/achromat.html

"Although I have acquired a thorough theoretical knowledge of the physics of colours and the physiology of the colour receptor mechanisms, nothing of this can help me to understand the true nature of colours. From the history of art I have also learned about the meanings often attributed to colours and how colours have been used at different times, but this too does not give me an understanding of the essential character or quality of colours."

Knut Nordby was in the same position as an LLM as far as colours are concerned. He learned all he could but words didn't give him any idea of *what it's like to see them". LLMs have access to everything that's written about colours, but that didn't help Nordby.

Nordby's no longer around to ask about image files, but how do you think reading thems would help? He might then have been able to say, e.g. "This pixel is orange," but he still wouldn't have known what it's like to see orange. The same applies to a multimodal LLM.

I'm not claiming that any particular disabled person has managed to achieve it. I'm asking on what basis you claim it to be categorically impossible.

You've claimed that black swans are impossible, I've asked how you arrived at that conclusion, and your answer is that you looked and don't see any.

You're here claiming that what submarines do is somehow fundamentally different than swimming and that we know this because they don't have legs. You need legs to swim. And I'm questioning the soundness of your reasoning and you're pointing to cripples without legs who can't swim as evidence. It doesn't make any sense.

> He might then have been able to say, e.g. "This pixel is orange," but he still wouldn't have known what it's like to see orange. The same applies to a multimodal LLM.

How is a multimodal LLM taking in RGB data fundamentally different than what your eyes are doing? That is the core question that you keep dodging.

> I'm asking on what basis you claim it to be categorically impossible.

Because the neural correlates of consciousness wouldn't exist in their brain. I've even looked into what happens if someone, blind from birth, takes LSD. There aren't any large scale studies as it's very difficult to get approval for them, but the short answer appears to be that they don't see things: https://www.livescience.com/62343-psychedelics-lsd-effects-b...

> How is a multimodal LLM taking in RGB data fundamentally different than what your eyes are doing?

The multimodal LLM is a program run as a process on a computer, which isn't aware of what it's running: it's just blindly, repeatedly, executing whatever instruction is at its program counter. The LLM itself isn't doing anything. It's just sitting somewhere in memory being executed by the computer. Brains, on the other hand, don't run programs. Each part of the brain has a special purpose, and somewhere in it there are neurons which correlate with what is being experienced whenever something is looked at.

> Because the neural correlates of consciousness wouldn't exist in their brain.

And what are those in concrete terms? Why can't a computer program have those? You have yet to articulate what special property is giving rise to this.

> somewhere in it there are neurons which correlate with what is being experienced whenever something is looked at.

And what exactly is special about the neuron that the biological hardware can do this but the inorganic can't? Is your claim that current silicon doesn't do this (but future might be able to) or that no silicon construction could _ever_ do this?

You have yet to clearly articulate what this magical pixie dust of consciousness is supposed to be.

> Why can't a computer program have those?

Because it isn't a physical object capable of sensing the physical environment, the way the brain is via the sense organs. I don't rule out the possibility that a robot, equipped with with a camera and microphone and processing the information it receives from them might be conscious of sight and sound, but not a program. If I read and execute a computer program in my head, I'm conscious of my thoughts while doing that, but are those thoughts themselves conscious?? I don't think so. What would they be conscious of, anyway?

> You have yet to clearly articulate what this magical pixie dust of consciousness is supposed to be.

Nobody has an explanation of consciousness, so the magical pixie dust remains a mystery. I think it's fundamental. Nobody has proven the opposite: that it arises from things (e.g. information processing) happening in matter.

An LLM is not a program, it's a static data structure consisting of a big bunch of numbers, and as such it cannot be run as it doesn't execute instructions or have a program counter. It is correct that the LLM itself isn't doing anything for this reason. LLMs are more like the neurons. The rest of what you're describing is not the LLM but the inference engine that does many lookups into the model and performs calculations outside the model. Brains do actually do the inference part as well as containing the data structure.

But you can't experience the qualia of tokens coming in and LLMs can.

I just explained this. If someone looks at the contents of an image file without displaying it, they don't experience the colours either. So how can LLMs experience them?

Right. Because your eyes ... imbue the magical pixie dust ... no wait, is it your visual cortex that does it? Hmm ...

Less facetiously, I'm unclear where you believe this magic "qualia" thing to be happening and what you believe the (meta?)physical mechanism to be.

Quite a lot has been published about neural correlates of consciousness, so I'll go with whatever the evidence suggests.

There are literally hundreds of attempted explanations of phenomenal consciousness, none of which is generally accepted. For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. At the very minimum, I think to be conscious of anything (other than nothing) requires the ability to sense the physical environment.

What's your position?

> For what it's worth, I think it's fundamental, i.e. not an emergent property of matter.

> requires the ability to sense the physical environment

Aren't those two statements at odds? Your synapses are made of matter, so sensing the physical environment is being done by matter. You are fundamentally matter (AFAIK). To suggest otherwise the only thing that comes to mind is substance dualism.

My position is that we don't know. I know I'm conscious. I assume other humans are simply because they're approximately the same as me and were produced by the same process. The farther you get from humans the less certain I am. Many insects resemble biological automatons if you examine them closely. Certainly single celled organisms don't seem likely to have anything going on.

If you figure that a clod of dirt can't possibly be conscious but a mouse is, what's the relevant physical difference? Electricity? Computers have that. Complexity? Information density? Both of those as well.

What do you think it means to sense?

> Aren't those two statements at odds?

Sensing (i.e. picking up information about its environment through sense organs) is a physical phenomenon and is reasonably well-understood. Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it. Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism.

So what's conscious? People and other animals with sense organs and nervous systems. It's even possible that anything which senses its environment could be conscious: https://organism.earth/library/document/what-is-it-like-to-b... . That might include a robot connected to a camera and other sensory apparatus. Now you could argue that a computer running an LLM can pick up what its user types, and might therefore be conscious of which keys are being pressed. But not of the colours, sounds, smells, and other things described by human authors in its training set.

> Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it.

Aren't you inventing that whole cloth? What evidence is there to support this claim? Can you empirically measure consciousness to demonstrate its presence or absence?

Is there a lobotomy you can perform to make someone unconscious but still able to respond to those around them? If such a lobotomy is within the realm of physical possibility then (at least theoretically) why can't LLMs be given equivalent functionality? If such a lobotomy is fundamentally impossible then by what mechanism is consciousness influencing human behavior?

> Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism.

So on the one hand you claim it's an extra (ie independent) step but on the other earlier you claimed that it requires the ability to sense the physical environment and that somehow LLMs can't do this (despite that I can feed them data from an environmental sensor). If it's an extra step presumably arbitrary data can be fed into it. You're simultaneously claiming that it's separate, yet somehow fundamentally tied to sensing the physical world (so not actually separate), and also that certain types of physical sensors are specially privileged (no physical mechanism proposed). And you don't seem to have evidence supporting these seemingly contradictory claims.

> Aren't you inventing that whole cloth?

I should have said, "I think that consciousness ..."

> So on the one hand you claim it's an extra (ie independent) step ...

I should have said, "The conscious experience (of qualia) is an extra step, and if consciousness is fundamental then that's a kind of substance dualism.

Thanks for pointing these out - I should have proof-read my reply before posting.

You still haven't answered the question. You think ... why? Based on what?

If consciousness is fundamental (ie substance dualism) then how is it exerting influence over our behavior? Isn't it the equivalent of watching a movie at that point?

If it's not fundamental then what precludes a computer program from exhibiting it?

Sorry for the late reply.

> If consciousness is fundamental (ie substance dualism) then how is it exerting influence over our behavior?

If I understand you correctly, it might not be exerting influence over our behaviour.

By influence, are you referring to free will rather than just the illusion of it? That's precluded by classical physics, and quantum mechanics just adds randomness. You can make a measurement, but there's no evidence you can influence the result you get.

> Isn't it the equivalent of watching a movie at that point?

Yes.

That doesn't really solve the problem. We can't conclusively say the models don't have qualia. Hell we don't know if a perfectly accurate atom-for-atom simulation of a human brain, would produce qualia.

I would (tentatively) agree — but that doesn't invalidate my argument: qualia are not objectively measurable, so you still can't tell if a machine is conscious.

Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1]

A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's many a people that have very different ideas and feelings on what's what.

The question on whether you can have "intelligence" without "consciousness" or "self-reference" are just on top of those. How can we define if a machine is "conscious" if we can't agree on what consciousness is? Panpsychists solve that problem by going the other way and saying that everything is conscious.

Overall these are very interesting topics that we can all ponder. The realities at the current time are, however, that LLM's are indeed useful. And the harnessing tooling that is being employed those days solve a lot of actual usage problems. Whether they solve other fundamental issues remains to be seem IMO.

[1]. I realize a lot of people probably haven't read the book, but you can ask your garden variety LLM to give you a quick rundown on the four camps that Sir Roger is using :)

Do you treat LLM as algorithmic? I mean, the process of computing tokens is algorithmic, also the process of training, but does it mean that LLM itself follows an algorithm, in a practical sense? If yes, how is it different to a biological system constrained by physics?

I would be interested in the answer to this question too. The visual cortex is certainly algorithmic in that light data enters, various cells and synapses react and fire and that information is ingested into something 'conscious'. Where does this then exist, and would it even be able to exist if we removed the biological algorithms at play?

A short disclaimer before answering. Right now I'm doing a Mathematics & Physics undergrad degree and have interest in philosophy. I'm not a specialist in any of those fields. I have plans to tackle the actual math around LLM's a bit further, but I'm not there yet.

To question 1 based on my current understanding (and let's say "beliefs") is that - yes, LLM's are following a probabilistic algorithm to give me the output. The same prompt won't yield the same result, but the same prompt will yield results that are quite adjacent to each other in meaning because of the training. There's randomness in the sense that you can't say which next token will be put with certainty, BUT you can be certain it will be a token that exists in the pool of tokens available. Hopefully that makes sense?

For question 2. Both us and LLMs are constrained by physics. The main difference for me is the fact that at every point of our existence we're constantly changing. Our internal state and bodies are changing based on all the sensory input we get from our environment and the processes that lie underneath. Another commenter in this thread talked about qualia. That's certainly part of it as well (but again we have a definition problem). That's not to say that I think that a hypothetical future machine cannot be devised to handle a lot of these, but the level of sophistication in the biological world is such that I don't think that's realistic. Let's say that even then, that happens, would that really prove that humans work the same way as the machine?

LLM algorithm does not need to be probabilistic, we add probability to make it more interesting, but in principle it should return the same tokens for the same input (which is often desired, and we had temperature=0 for that, even if for practical reasons it was not always working that way). But it is not the point. My point is that LLM inference follows an easy algorithm, but what we get is something different, because weights are part of the algorithm and they are not easily interpretable. So if I ask LLM to write me an essay on a given subject I know what it is technically doing, but writing it as an algorithm different than "convert this text to tokens and than perform billions of simple operations on them to get the next token and repeat" seems hopeless. Because the black box of weights is what matters. In this meaning it is not an algorithm we can find in a book on algorithms. Of course it's a computational process, but the question is, if it is qualitatively different to the one in our heads.

Do you distinguish between Penrose's argument and the older one by Lucas, invoking Gödel's proof that certain facts are non-computable?

I always felt it's basically the same argument, and Lucas's was never really convincing; Hofstadter rebutted it very convincingly in GEB long before Penrose's books came out.

To me, camp C's claim is possible but I don't buy the argument that says it's necessary. So I guess I'm camp A. (D is religion and B seems to me incoherent.)

I haven't read GEB (yet), but I'd say I do feel there's a difference between the original formulation of Lucas and the extensions that Penrose provide. At this stage I can't say whether I'd agree with Hofstadter or not on the rebuttal though.

As for the camps, I agree with you that B doesn't really make sense (at least to me). I can put myself in position to argue for A, C and D, but as mentioned in my original post so far I agree with C the most.

It “throws” an error when “it tries” “to talk” to the “server.”

We’ve been speaking as if the computers were conscious for a long time already.

Our definition of consciousness requires stakes.

We associate consciousness with precious life: Life that comes with rights, needs, wants.

Life has rights because it naturally demands to be continued. If it makes no difference whether it is continued, it needs no rights, and we do not consider it conscious.

Life has needs because it cannot survive without. If it has no needs, it has no death. No death, no life.

Life has wants because it is harder without. If it has no wants, it has no emotion: It cannot feel pain, otherwise it would want no pain. It cannot feel happiness, otherwise it would want more happiness.

We don't want to build consciousness! :) Deliberately designing pain and artificial needs is a cruel exercise. We can build that, but we see no need to.

The real question at the table is, are these minds? And can minds exist without consciousness? Unequivocally, yes.

I think this is exactly the same problem to decide whether any human is conscious, just because they say they are conscious. Humans can act as if they are conscious too, they can even lie to themselves, their whole life, and never discover the truth about their own mind

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?

Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?

Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.

When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...

What exactly is the fundamental difference that creates this subtle distinction?

Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.

The fact that we constantly switch between the colloquial definitions of those terms and more specific, technical ones doesn’t help

I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following:

The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)

The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)

I dont think Ill fire the human nor the agents. Both bring a lot of value

I would be extremely offended if I heard someone claim they "prompt" me. It's very disrespectful.

It was obviously meant as a funny joke ;-)

But on a more serious note. Humans "prompt" each other all the time. Especially in a Boss->Employee relation.

Defintion of "to prompt":

1. To cause or inspire: To make something happen or motivate someone to take action. For example, a loud noise can prompt you to look outside.

2. To assist or cue: To help an actor or speaker remember forgotten lines or words.

there is deeper thought on this topic beyond douglas hofstadter (~20 years ago for strange loop and prior for GEB) and scott aaronson

any cognitive science textbook will bring some thoughts, or a cursory google scholar search

I found that in LLMs the self-reference defined as "referring in generated text to itself" (so a bit different than in Scott's blog) comes largely from the chat template

https://openreview.net/pdf?id=3O2A23MhNi

An incrementally built response seems self referential to me.

I care more about the recursive loop between execution and cost. Anything that executes - be it biological or artificial - incurs some costs. Those costs in turn impact what execution can follow. The gains pay for action, action produces gains.

LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.

LLMs feed on their own output during inference, so there is a loop on itself, isn't it obvious?

An LLM is no more self-referential than a PDF reader showing its user guide.

It seems like it is the bitter lesson of the emergence of intelligence.

Yeah but that is the whole point, right? Maybe I am thinking too simply, but it seems obvious to me. LLMs are intelligence without consciousness. Once we build in a self-referential learning loop into the models, it will gain a form of what we consider consciousness.

When you put it like that, I think there are actually three separate things we're arguing about:

- intelligence: ability to solve problems

- agency: ability to be self-directed, to choose what to do

- consciousness: ability to experience things; "having a self"

The "intelligence" part is largely solved now (that's a very big statement, but I can't think of a better way to phrase it!) Although really it's just moving the goalposts -- computational stuff like calculating trajectories or playing chess was solved long ago, this is just more and more things moving into the "solved" column.

That leaves agency and consciousness as the hot topics that nobody has really figured out. Do they always go together, does one require the other, does one create the other? Who knows?

"Free will", there's another one. That must surely connect with agency, with consciousness, or both, in some manner we haven't figured out.

To me, agency seems like a solvable problem via existing approaches and technology. I have no idea what that means for consciousness. It's tempting to conclude that consciousness is just a mirage, but then we all do feel like we have it, so it seems like it must be something. Maybe consciousness arises automatically once you have sufficient intelligence and agency; but how would you ever determine that?

Other people might think that true agency requires consciousness, and that consciousness requires some magical new ingredient that LLMs don't have yet. The problem with that approach is, you have to identify what it is that true agency can do that software can't do; and every time you do that, it turns out LLMs can do it, so you have to keep moving the goalposts. Trying to make your argument rigorous immediately makes it self-defeating; I think that's why so much of the philosophical discussion around this is impenetrably vague. All the arguments that aren't vague just turn out to be wrong.

I think the missing idea for AI is in Iain McGilchrist's The Master and His Emissary. The book's thesis, very roughly, is that the hemispheres aren't redundant processors: the left specializes in focused, decontextualized representation (sound familiar?), while the right maintains broader, contextual engagement with the world (McGilchrist fuzzily reduces it to "what is out there?" which is fundamentally about ascertaining uncertainty as a mode of attention). And crucially, the corpus callosum connects the two processors and crucially helps preserve their functional separation through certain channels (as both sides are connected).

Transformers look suspiciously like very powerful left hemispheres. They are phenomenal at manipulating representations, but increasingly weird when the representation gets mistaken for the thing itself. Hallucination, sycophancy, context collapse, overconfident completion, etc. all look less mysterious when you compare it to the lists of pathologies that those with right hemisphere injury face (e.g. anosognosia, confabulation).

So what seems missing isn't a Gödelian strange loop but the other side of the brain and a mediator between the two sides.

The way I think about it is like Kriegsspiel, the ur-game for roleplaying and wargames. To truly "simulate" war (or a fantastic medieval adventure game), the simulation requires three sides: The Blue Team with a goal, the Red Team with its own goal (typically to stymie the Blue Team), and the Umpire, whose only job is to objectively simulate the effects of the orders of the two sides.

The Umpire has the true state of the world; Blue and Red only get observations through the Umpire. Neither can directly inspect the other's state. Each side must provide orders to the Umpire, who, ideally, being an SME, is able to process them both according to the rules of Kriegsspiel but also their real world experience in war. And that's the whole point of it in the first place: the Umpire maintains the fog of war between the players, with the ultimate purpose being improving the generalship (i.e. learning) of both the Blue and Red sides. And what is generalship really? It's being able to discern resources and operations in a world of Knightian uncertainty where one must evaluate whether there is an active intelligence attempting to thrwart your model of the world and your goals.

(An aside: I think this is why Dungeons & Dragons fails fundamentally as a rules/rulings-bound game. The dungeon master and referee are the same person, while each role has a fundamentally different goal. With split roles, the DM can actually try to kill the players using the Dungeon with the same level of asymmetric information as the players and the GM to actually referee the game neutrally with a view to what actually happens.)

Nobody actually gets the God's-eye view. The Umpire gets objective reality but lacks insight into the minds of either side, both of which are pursuing unique ends. This triad allows for genuine Knightian uncertainty in games where judgment (i.e. creating and using precedent) is fundamentally required. This triad is also shared by law, government (and from my perspective trinitarian theology). Intelligence becomes qualitatively more powerful when the architecture prevents any one component from possessing a complete, self-consistent description of the whole, while allowing the components to interact through constrained third parties.

I suspect this is also why human cognition seems to somehow avoid Godelian incompleteness by sidestepping the requirement that, as a formal system, it contains a complete model of itself. Just give the system multiple partially informed perspectives coupled through an epistemic boundary, and incompleteness becomes a source of entropy sifting and uncertainty rather than failure modes.

And if what I suspect is true, at some point the qualitative bottlenecks that pure transformer-style systems possess should begin to disappear when this approach is applied.

But I'm generally an idiot, so take all this with a huge grain of salt.

> What can we say about this with hindsight? > While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative.

The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.

Weird side swipe at Marx. Or not so weird I guess. Correct me if I'm wrong but I doubt Marx ever predicted that a workers' uprising would lead to classless utopia. It's clearly a necessary but not sufficient condition.

Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.

"Understood — I am no longer using any em dashes"

That's just over-fit. Happens in humans too - like an alcoholic might swear to stop drinking.

It was demonstrated that LLMs are capable of non-trivial self-introspection. E.g. if a steering vector is injected into residual stream, a sufficiently large LLM might be able to describe what that steering vector represents. A fine-tuned model might be able to describe activations, etc.

Reply on news.ycombinator.com