Comment by perching_aix | Hacker News Reader

Comment by perching_aix | Hacker News Reader

Parent Full thread

Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*).

(*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").

You'll sometimes catch models mixing up who's who and how many who-s there even are for example.

(**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn.

I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't?

(*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".

Before reading, know that I am uncertain in either direction.

> a hidden representation of self that is continually tended to

This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it.

> I further fail to identify how it could be hidden or maintained, considering I control like half of it.

Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output.

> The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").

While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one".

> You'll sometimes catch models mixing up who's who and how many who-s there even are for example.

I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that.

> If you know of anything like this, your turn now, would be happy to learn.

Do you mean like these, or something else?

• https://researchportal.hkust.edu.hk/en/publications/decoding...

• https://aclanthology.org/2026.eacl-long.165/

• https://transformer-circuits.pub/2026/emotions/index.html

> This sounds like a personality?

Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions.

> Perhaps they were losing their self-awareness at the time?

I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way.

> Do you mean like these, or something else?

They're adjacent, but I more meant something like these:

https://arxiv.org/abs/2410.03768

https://arxiv.org/abs/2310.18512

https://arxiv.org/abs/2605.26537

So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state.

And the best inkling I have towards this is basically: https://www.youtube.com/shorts/WP5_XJY_P0Q

But then I don't think there's enough covert channel bandwidth in the agent replies for anything interesting like this.

> Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to

I don't see why an LLM could not have a sense of identity or personality while it's evaluating a specific prompt, or even change self awareness while evaluating a prompt since many outputs model a back and forth conversation. My point is that without a mechanistic model of what "self awareness" means, we have no way of truly evaluating such questions, we're just hand waving vague intuitions about what it could mean.

Sure, but then such a model is not going to make itself. People pitting their vague intuitions is how such models eventually form. I'd also push back regarding that my comment would have been handwavey or without mechanistic elements, even if it was on the whole informal.

This is kind of also the reason e.g. the HN site guidelines are worded the way they are. Regrettably, forums naturally yield themselves to tit for tat type exchanges, but there's really no reason one could not bounce such vague intuitions off of another. I do not have to be right or wrong, and you don't either. Admittedly difficult when its some intensely contentious topic.

If a mechanistic model existed, there would also be no reason to talk about this in the first place. There'd be nothing to discuss, you'd be simply told how a given model characterizes from this perspective on the model cards.

Even mechanistic models generate interesting discussion. How many years have we discussed Turing machines and the lambda calculus? Almost a century of great work came out of those.

The reason I insist on mechanistic models is because the original post was making a definitive knowledge claim, and in my experience, the knowledge claim is unwarranted.

Reply on news.ycombinator.com