LLMs and Self-Referentiality

> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity. Not the self-reference stuff.

> What’s left? Consciousness and subjective experience of course remain extremely mysterious. For all we know, Hofstadter could be right that those have something to do with self-reference. (For all we know, even Penrose could be right that they have something to do with exotic physics accessible to biological brains but not digital computers!)

I'm not convinced that this is the case. Biological entities have a spatial-temporal sense, a memory, and, alongside those, a predictive/simulating function in space and time. A cat who knows where it is in the world at a given time, sees a mouse, and predicts where that mouse is going to go, is essentially performing a physical calculation, much as you do when you attempt to catch a ball. The more [powerful + general] the predictive function, the more intelligent a given biological entity is. Past a certain threshold, the predictive function becomes capable of broad abstraction. This enables man to acquire, process, store, represent, and continually re-acquire (in a Bayesian sense) knowledge that is external to that man's subjective existence. This is pure calculation in the strictest sense, and doesn't require exotic physics or self-reference beyond a sense of being bounded in space and time, which is to say a reference point.

All that aside, Hofstadter and Chomsky have been pretty hard-hit by LLMs!

Then where does self-reference appear in other animals? Famously most higher animals such as mammals still fail the mirror test.

I agree with the point that self-reference comes "for free" due to computationalist reasons, namely Quining and Godel and liar's paradox all show that. But then self-computations should be more prevalent relative to phenomena like consciousness. Example could be the human immune system functioning, in that on the level of DNA there is some encoding of identifying what is "self", etc.

Also if self reference is easy for the reasons he mentions then should we see non-LLM systems, say AlphaGo or image generators, learning a sense of self vs other? E.g. inside AlphaGo's black box it could utilize a concept of "my strategy".