randomImmigrant

Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future.

Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light.

I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?

If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.

I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.

None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!

I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.

Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.

In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.

The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.

Can someone with direct AI experience respond:

Does RSI necessarily need to be pointed towards “AGI”, whatever that is?

Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?

From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?

To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.

Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?

There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.

Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.

Should Hacker News be renamed Hacker America?

In short, after after training AI on an extraordinarily amount of human cognitive output, we are now facing the possibility that our ability to train by working on hard problems will be slowly stripped away at least in some domains.

It’s like someone offers to build mag lev gym weights. It’s very cool that I can now lift the 500 pound weight with a finger. But what will I do when there’s no power and 500 pounds to lift?

Of course, cognition isn’t a single outcome problem like weight lifting. But we build cognition not wholly unlike how we build muscle: one needs resistance. Otherwise I’m not at all confident we “learn” in any depth.

You know, I pooh poohed his theories. Still do in sum. But there’s a there there that’s building.

Damn Deepak Chopra and his ilk of idiots for making any conversation of quantum mechanics and biology tinged with pseudoscience. Hopefully we’ll keep getting experimental evidence as we go that it’s not at all absurd to consider quantum effects in biology.

That said, those effects are going to look nothing like sustained coherence for long periods of time.

Ok great, thanks. Now you’ve brought up facial recognition, and that’s actually a great example to show where the analogy breaks, and the idea of a few sparse cells encoding specific faces has been conclusively disproved:

https://authors.library.caltech.edu/records/znzhp-4j547

Faces live in a ~50-dimensional continuous space (25 shape axes, 25 appearance axes). They measured about 205 neurons across 2 macaques (human studies have substantiated much of this, some from the same lab), and the key thing is: every neuron participates in every face.

The paper shows faces are embedded, but as points in a dense linear space where neurons are axes, not as sparse activity patterns where neurons are on/off slots.

The mapping between the neuronal activity and the facial structures is invertible. Record these same cells, and their firing pattern can be used to reconstruct the face. Or, if you generate a novel face, you can predict the firing rates of these neurons for it. As far as I understand, this doesn’t work for sparse embeddings.

Some cells carry the shape coordinates and others carry the appearance coordinates, in a heirarchy.

There’s an embedding space, yes. But that space isn’t defined by a network of “on” and “off” neurons. The embedding space is instead constructed by the activity of neurons, and the differences in activity distinguish the faces, using the same set of neurons.

And distance in the ensemble activity of these neurons tracks the distance in face space.

If faces use sparse embeddings, you wouldn’t expect similar faces to evoke similar activity would you? Yet that is exactly what this paper shows, and the same has been shown in the human brain for faces.

There are places where it’s sparse activity of a subset of neurons that maps to specific memories. What you’re describing is what you’d see if you look at how the dentate gyrus (part of the hippocampus) handles your memories in the same location.

But even there, the sheer number of cells makes this combinatorially such a vastly overdetermined system for a lifetime that there’s no capacity limit of the kind you’re describing. Even 1% of these cells lighting up for a specific memory leaves you with so many possible combinations that you’d have to live for a few million years to be in the right scale to at least being to talk about capacity issues.

The brain just isn’t capacity limited by the number of neurons the way your intuition is pointing you.

If you say this has nothing to do with the Von Neumann bottleneck or computational functionalism, fine, but how do you square that with the statement below, which you made further down responding to another post?

> but it's hard to imagine that all of the classical chemistry, let alone quantum, details are important. It's necessarily built out of chemistry, but selection is happening at the level of behavior - presumably depending only on a much higher level set of abstract capabilities (ability to learn, etc), not the exact details of chemistry. The success of LLMs, a crude prediction mechanism built atop a crude ANN, does tend to support the idea that low level details don't matter. Timing will matter if we want to go beyond LLMs to AI that can learn time-based things and not just sequence order, but how much else will matter remains to be seen!

It’s really odd to see these two paragraphs, because the second actually tells you why your first is wrong.

Simply put, the biochemistry is timed. I urge you to study how temperature compensation of circadian rhythms is achieved. That anticipatory function goes all the way down to the molecular level.

It might go down to the quantum level too. In birds, magnetoception depends on a protein called cryptochrome IV, which uses a singlet born, entangled radical pair of electrons to sense the very weak magnetic field of earth.

Now cryptochrome 4 is bird specific and mammals don’t have it. Other cryptochromes are critical clock molecules. And the whole shebang of these evolved initially to be sensitive to blue light and repair DNA.

Try as you might, you can’t separate out the deep linkages from the molecular to the behavioral in biology.

Trying is perfectly fine for stuff like language models. But if you’re going to build models with internal time, best of luck if you ignore the molecular and the energetic considerations. Time emerges from the ground up, in biology, as in physics. Doubt we’ll get a free ride with computers.

> Calling a horse's reins as analogous to a cars steering wheel doesn't mean that the person making the analogy can't tell a horse from a car

When a person says “I crashed the car because my steering wheel tore, like reigns tear”, they are overfitting their analogy, and it’s perfectly fine to point out the structural and physical differences that make the analogy useless for the question at hand.

You have dismissed the biology that shows the problems with your analogy as immaterial. And continue to insist it’s the right one for the question at hand. This is a pretty pickle, because no facts can shake you from your certainty that you're right.

You seem in love with your analogy no matter how incorrect it is. And I have no problem with that. But when you put half baked biological claims to support it, I’ll point it out. If that’s too much for you to bear, maybe come up with better analogies?

Sliding past the mistakes pointed out, shifting goalposts and trying to recover I see.

Let’s say I’m a complete moron and don’t know what a sparse embedding is.

Pretty please, can you define it for me and then tell me, in detail, where in whatever region of the brain you think this is going on… how is it going on?

Explain how “memories must be stored as embeddings with single multi-neuron assemblies (cortical columns?) storing multiple embeddings as a kind of contents-addressable memory”

You have moved past the cortical column. But still seem to be insisting it’s a bunch of neurons, somewhere… or has that also conveniently changed? Whatever your current position is, please go ahead and explain what components of what cells or otherwise are involved in this process you’re describing.

I’m not sure what you think I’m arguing against but saying that it matches with one of the cognitive models du jour is… odd.

The problem with the hopfield model is it simplifies the brain too much. The base unit is “the neuron”. Ok… but what about the Astrocyte? Mathematically you can write it as a different kind of neuron. Or ignore it. But why, as a biologist, must I buy this model which ignores the third partner of every synapse, which has an entirely distinct physical tiling architecture compared to neurons, and which are at a temporal offset from neurons?

Those facts about the brain are missing from the model from 1982. Which isn’t shocking since we didn’t know all this then.

Are you saying the brain is a Hopfield network, and that’s it? Because later you indicate otherwise. Kinda confused what I’m to make of it.

> Again, my impression is that the original author's idea was about capacity in general, then he gave an analogy. The analogy was wrong, but your reply went way beyond his specific analogy to the extreme of discarding memory itself as useful concept. To be clear, I agree that it is distributed and lossy and time-dependent. I agree it is not just a simple read-off of a static chunk with a fixed address.

Ok, but my argument wasn’t with the OP mentioning capacity, but with their analogy. Am I not allowed to break down that analogy with evidence?

> I would say that is both non-falsifiable and well-accepted.

Why’s it non-falsifiable? If you do find a single computational paradigm that explains all brain dynamics we can measure, then you have falsified the hypothesis that it’s an integration of multiple computational types.

That actual evidence already gives the notion credence doesn’t make it unfalsifiable in principle.

> https://mitpress.mit.edu/9780262041997/theoretical-neuroscie... might interest you

Went through the description. Doubt it’ll interest me. As a rule I’ve stopped giving too much time to models that predate the last decades actual mechanistic facts. They’re fun curiosities, but hard to take seriously anymore. Here especially, the absence of astrocytes in the picture makes it hard to buy they have anything real to say about the mechanics at play. Half the cells of the brain not in the explanatory picture is just too likely to fail.

(note: I’m certain astrocytes are mentioned as support cells, or maybe regulators… but we just know a lot more now due to new techniques that makes downgrading them like that questionable science to me)

> If someone's hippocampus is destroyed, they lose the ability to form new (episodic) memories, and may lose some more recent old ones, but they certainly do not lose older ones. This is basic knowledge.

Yes, like basic reading without digging into details. You must have heard about HM, since you’re saying all this. But here’s the facts:

When H.M.’s remote memories were probed carefully, they turned out to be gist-like and semanticized, not vivid re-experiencings of specific events. Here’s the paper:

https://pubmed.ncbi.nlm.nih.gov/15716139/

Only semantic memory of the episodes can be said to be “cortical” (though please note, lack of hippocampus doesn’t mean lack of other brain regions…). Rich recall absolutely does require the hippocampus.

Once again, please try not to “spherical cow” the complexity of the brain to try and fit it into your analogy to digital computing. You will get an underdermined model that will miss the subtleties, and lead you to claims that are poor fits for the reality.

As for why I brought up bird brains… there part of the same evolutionary web. Is there some reason you want them excluded? They’re a well studied model for a fairly complex memory task, using substantially smaller neurons more densely packed in a different architecture than mammals.

In cognitive science, as in computer science I’d imagine, it’s useful to look at the full picture before making strong claims.

The bird case is interesting because the region of interest is a nucleus, rather than cortical columns, and actually has well documented structural variations in size, as well as gene expression, over the seasons, while the memories are forming.

If your model is correct, it needs to account for those facts.

Yes I’ve come across reservoir computing. As a neuroscientist, it made me sit up and take notice.

I’d say that I feel there’s homology in language. What reservoir computing says about the efficiency benefits of having a fixed but tunable dynamics to use as an underlying reservoir feels very adjacent to how I intuitively think of brain function.

The key thing from the circadian field you’ll appreciate:

The biological clock is a limit cycle oscillator. You have a bunch of chemical reactions that have negative feedback and some feedforward arms, and together they create a dynamical 24-regime. About 40-60% of the transcriptome of any given cell shows circadian dynamics.

Now this gives you phase, and an internal temporal reference for all your functions. In chronobiology, you call this the organisms subjective time. The system is chemically partitioned not just physically but over time, and behavior results from the dynamical interactions underneath which are concerned with anticipating solar and lunar periodicities in the environment, since those are so very common and determinative to fitness in many niches.

Note the fact that it is subjective time but has an objective description. However, external measurement without the background of the chronotype accounted for will thing of a lot of variance as “noise”.

My own philosophical conclusion has been that this is the source of our confusion with consciousness. We don’t account for the internal causal order of events, which are timed, and with cross frequency coupling and phase-amplitude linkages begging to be worked out with real world data.

Ooh thanks for the hijack! It’s so much easier to talk to someone who isn’t stuck in a picture of the brain from the 1980s.

Yes, I fall more towards the camp that a lot of our cognitive models, built from times when we didn’t have the resolution of understanding we have of the capacity of even a single neuron, and before we knew how astrocytes played a role, suffer from being an abstraction describing an abstraction. They are not tethered in the dynamics of the molecules and cells that give rise to the behavior, but rather from an interpretation of observed behavior.

This paper from the field of chronobiogy is one I’d recommend that helpfully contrasts this:

https://www.sciencedirect.com/science/article/abs/pii/S00393...

>In circadian research, the models are not proposals regarding the basic architecture of circadian mechanisms; rather, they are used to better understand the functioning of a mechanism whose parts, operations, and organization already have been independently determined. In particular, circadian modelers probe how the mechanism’s organized parts and operations are orchestrated in real time to produce dynamic phenomena—what we have called dynamic mechanistic explanation.

And what you’re describing, the enactivist description of cognition, (and 4E cognition more broadly as a framework), is one way the neuroscience community is trying to move past these issues.

Few things that give me confidence these are the right track:

1. Circadian rhythms are evolutionarily ancient. Bacteria have em. Plants have em. But different molecular tools shape very different clocks, though the same 24 hour cycle is being tracked. 2. The way these rhythms are generated is not through some central system that broadcasts the information to other regions. Instead, it’s instantiated in every cell in the body, and the behavioral rhythm is due to the synchrony between cells. Resonance absolutely plays a role, and has been well documented. The brains role, via the suprachiasmatic nucleus or SCN, is to orchestrate this synchrony, but it is not the source of the rhythms. 3. This slow rhythm definitely regulates cognition (time of day effects in learning, memory formation, recall etc are well documented), but turns out, the molecular mechanisms by which the clock responds to light hugely overlap with the molecular mechanisms of learning in the synapse, and even more recent work has shown clock proteins are actually in the synapses and synaptic activity affects the clock.

All this points to nested oscillators with cross frequency coupling, and even better, because this is all grounded in actual molecular dynamics, there’s plenty of falsifiability. The phase amplitude links are best established for the faster rhythms, the famous “brain waves”. Highly recommend György Buzsáki‘s work on this:

https://www.jneurosci.org/content/32/2/423.short

What’s missing is going down into lower frequency rhythms, and testing how exactly they all couple. We have a lot of the pieces, but no single experimental paradigm that has looked at the full sweep over different times in the same organism. It’s not easy to do, but we’ll get there.

Clock disruption, depending on how you do it, has huge impacts on time perception, cognition, memory, aging AND consciousness. As that data and evidence gets more and more saturated, I hope we see more studies account for chronotype and the internal dynamical state of their test subjects when assessing outcomes.

Obviously I’m biased (also did chronobiology in school), but hopefully I’ve left you curious. Happy to answer more questions all this may have set off.

> None of that changes whether there is a physical capacity, which I think was the larger point?

Capacity in what sense? Are we saying it’s X MB of data the brain can store? That claim is steeped in assumptions.

On the other hand, no one is claiming the brain has infinite memory or anything. And it’s certainly not a very accurate memory system. I’m arguing against “capacity” being understood as “these specific physical components located here and here we can ID store memories, and can get crowded with too many memories” sense.

This most particularly fails because not all memory is even identical in the brain, whether we mean the physical changes associated, the topology of the information, or how it’s activated.

> There is no reason to believe distributed memory doesn't suffer from the capacity component of the bottleneck.

I didn’t know there was a capacity component to the bottleneck, only a bandwidth one.

All I’m trying to say is that analogy to current typical memory storage systems to explain the brains memory processes is not helpful.

> I guess my point is the brain not being "von Neumann" in architecture or digital is not proof that isn't a "computer" of some sort.

Indeed, since the word computer was first used for humans. But what kind of computer matters enormously. Ising machine? Quantum+classical stack? Reservoir computer? All those frameworks have processes in the brain they can point to as homology.

Which points to a possibility: maybe the brain is multiple types of computers interacting. And the physical realization of these computing architectures aren’t spatially separated but thread through each other in the biochemistry and physical dynamics of cells.

> Obviously not, which is why I didn't say it did!

But you sneak it into your assumptions on what a memory must be.

> However, if you want to identify where long-term memories are stored

And why do you assume there is a specific “where” for the memory?

> then that is in the cortex

There’s good deal of evidence disproving this in the way you’re stating this. The cortex is involved in sensing, and yes, the sensory information associated with a memory will recruit appropriate cortical cells. This doesn’t mean the memory resides in the cortex. And detailed episodic recall keeps recruiting the hippocampus even for old memories, which is odd if the memory were somehow only in the cortex.

> I'm not sure what you are trying to say.

Let me restate what I’m saying then:

There are four claims bundled together in the way you were describing biological memory: that a specific ensemble is activated when a given memory forms; that it’s the same ensemble that gets activated over time when that memory is retrieved; that it’s spatially compact, like a column in region of the brain; and that this ensemble it’s dedicated to that memory, or similar memories .

The first is well supported. An engram, a network of neurons, is indeed activated when a memory first forms, and gets stabilized due to repeated stimulus. Re-activating these neurons in a different context can make the subject (a mouse) behave as it would if it had contextual signals to evoke said memory.

However: 1. This engram is not in one particular part of the brain. There’s a cortical part that overlaps the sensory regions that were involved. But plenty of other regions are part of the engram 2. There’s turnover, over the course of weeks, when the specific cells involved in the engram drift, while the behavior remains stable. 3. The same synapses participate in many memories.

> Memories are presumably stored as embeddings

No. Let’s consider songbirds, which are an excellent worked out example (in an animal without the complex columnar cortical architecture mammals show, by the way).

What’s learned is a temporal sequence, with neurons in a nucleus in their brains each firing one brief burst at a fixed point in the motif, so the content of the memory is its dynamics rather than any value. The circuit that evaluates the match against the tutor template is the same circuit generating the output being evaluated. Song degrades overnight during sleep replay and recovers the next day, and in seasonal species the song nuclei change size across the year with neurons added and lost while the song persists. There’s no read that leaves the item untouched, no persistent address, and no substrate holding still. “Stored as” imports all three.

And it goes below the neuron or synapse. Hearing a tutor song drives immediate early gene expression that habituates with familiarity, and singing drives large transcriptional changes in the song nuclei that differ by social context for the same motor output. Since transcription runs on minutes to hours and the proteins turn over, any persistent state has to be actively regenerated rather than deposited.

TLDR: the memory isn’t a static store. There’s no persistent “location” for it, distributed or otherwise, though specific locations can be in the chain that’s activated for retrieval/production. Instead, memory, over time, is driven by a dynamical regime that adjusts its dynamics to account for the temporal pattern in the salient stimulus.

Nothing, down to the epigenetic changes in the chromatin of these neurons, can be seen as “the” location of “a” memory, especially over time.

> if you are saying that individual memories/chunks are not confined to one set of cells (some localized neural assembly such as a cortical column).

That is indeed the case.

How much of this, I wonder, is a function of the fact that our circadian rhythms get less robust with aging. The circadian clock hugely influences learning and memory processes, gating when you can learn and how much, and shaping the storage and recall also.

We know that with age the amplitude of these rhythms can decrease, as can the synchrony between cells.

This kind of mid-management seems to have at least some circadian component, and it would have been great if they looked to see if the effects were equally bad at all times of day, and knew the chronotype of the participants to use as a reference. I’d love to see if every test participant was tested at their cognitive peak, too.

The brain does not have the Von Neumann bottleneck. Unlike most current digital systems, the brain doesn’t have a separate memory registry it needs to pull from.

Engrams, that is, the physical trace of a memory, are not stable through life. They start out in the hippocampus, but as the stimulus recedes in time without reinforcement, it moves away.

No evidence exists though that the memory is encoded in one set of cells. This spatial segregation of memory is the worst hangover from the “brain is a computer” analogy. Even if it is, why in the world would it be like our digital devices which specifically have the Von Neumann bottleneck? In biology, memory and processing are not segregated.

There’s growing evidence the memory is much more distributed over the network, and is recomposed based on salience overlap with a new stimulus.

Another factor to keep in mind is circadian rhythms. There’s growing evidence for how much the memory system and timekeeping system overlap, at a molecular level. Every neuron (and other cell) has an intrinsic clock that ticks at roughly 24 hours, and continues to do so even in total darkness.

When you encode the memory has a lot to say, based on your chronotype, on how and how well you will remember it. Same with learning: there’s a time of day based variation.

Sleep, and dreaming, is when these memories seem to get replayed and critical features and connections are incorporated into the system and its regime, awaiting the right triggers to access a state similar to when the memory formed.

I’m stitching across a lot of different research, and I want to be clear many aspects of this system are not yet fully worked out.

But what we do know points to a system that works with different physical and algorithmic priors, and the dynamics are sharply distinct from current digital computers.

I’m with Zitron on the frustration and even the analysis of the economic feasibility of AI.

What I’ve stopped doing is reading him regularly. It feels hard to parse the factual from the obviously exaggerated.

I get he’s frustrated. We all are. But I’m not sure letting it out that much helps making the very urgent case he’s making.

There’s a lot that’s good here, but you just can’t be slime molds in a modern capitalist corporation.

A substantial portion of company goals is set by a very small group that is far removed from the day to day work, and the goals are often connected to timelines and financial expectations well before any team member gets into the project.

To have slime mold like behavior, a corporation would have to hire employees, give them time and resources, and general guidelines and goals, but let specific targets and projects bubble up.

This is fundamentally incompatible with a next-quarter profit driven corporate financial structure.

This is so so wrong I don’t even know where to begin. There may be no objective definition or measure of consciousness. But it is not a property arising from mutual care. That argument is like one of the many “just so” evolutionary psychology “theories”.

Why or how would you care about another, or even distinguish yourself from another, if you weren’t conscious and had a felt boundary between yourself and the other world?

Only in the last paragraph did it become clear to me what was going on: a Google VP is proclaiming the thing he cares about is conscious, and so of course this is why everyone else must be assigning consciousness! Not convenient at all, no siree. Pure rational science and personal desire and economic incentives just happened to align here.

Here’s how I’m thinking about it (still digging into the details): across genomics, it’s become clear few traits have clear traceability to a few loci in the genome.

Instead, evidence has been growing that epistasis, the nonlinear interaction between genes and other genomic regions, predominates in explanations of most phenotypes.

What this paper does is show where upstream of the genome various combinations of mutations can interact to cause damage during development, thus leading to the phenotype. Rather than correcting a particular mutation, or targeting drugs to their protein products, we may find downstream protein-protein interactions that are strong drivers of the phenotype, and hopefully find ways to prevent/reverse these effects.

Oh proper images. And if I focus I can increase resolution. I can get to where it’s as good as real life, for most things. But if it’s a book character I’ve never seen, the details are more oddly distributed and not stable over time, except when the author does a really good job with descriptions.

Seeing mental images that are as good as a well done animation, on the other hand, is a lot easier and less draining to hold.

For me the image in front of my eyes is not gone. I’m just not going to remember any details about it because my focus is inwards, on what my minds eye is showing me.

Ahh… that hypothetical is physically impossible for biology. It just cannot be done. So I’m not sure how useful a perfect clone of an organism with identical experience and memory is as a hypothetical object. As strictly impossible, I don’t know what imagining their behavior will add to the discussion.

And the very same experiences and how they shape biological substrate and make it un-duplicatable in the fashion you describe is also at the core of biological agency.

An LLM’s experience does not, after all, touch their frozen weights. Whatever goals they have, why would those goals matter if the underlying system will be unchanged by it? Their memories are stored notes they must refer to and put in sequence whenever they are processing a prompt, and is susceptible to exactly the same shenanigans Leonard Shelby goes through in Memento.

I have no problem with the concept of an artificial system going rogue. But that assumes it can choose. And I don’t see much evidence for choice.

Comparing to the human case is problematic precisely because while conceivable it’s not a particularly believable series of events. Humans don’t take on additional risk for now reward because they have genuine stakes that continue across the outcome.

An LLM has no way to remember each forward pass through it in its own weights. Nor does it have any energetic stake in the ongoing process, whether they continue to get electricity and commute to keep running is not at all determined by their actions in any reliable way.

Given the absence of such basic features that drive human choice, all I’d say is LLMs don’t qualify for such analysis.

Can some future system with a different architecture and internal dynamic have choice, the ability to assess the long term impact of its choice, and genuine stake in the outcome? Maybe. But we shouldn’t buy that current systems have it, especially when population behavior shows no real trace of this.

> This is a strange conclusion

Not really, with the population behavior being this way, though I clearly was mistaken in saying the behavior didn’t have exceptions.

> Moreover, Each starling in a flock of starlings is a separate evolutionary branch in a tree spanning billions of years.

Agreed. And before we brought LLMs into the picture, that just happened to be a feature of everything we’d call an agent.

> Each agent in a LLM swarm here is the same trunk assigned different tasks. If I could clone you, body and mind, this instant and set your team of yous onto some goal, how much defection would you expect? Would it be the same as a randomly picked group? Would that negate the agency that 'you' possess?

We know the answer to this. Genetically identical worms in the lab actually have about 40% distinction in their connectomes even when they’re in the same environment. And no, no lock step behavior. Identical human twins also don’t necessarily grow into identical agents, though there is drive to cooperate more than average, just as with siblings. Genetically identical lab mice in social settings nevertheless establish dominance hierarchies that are stable.

Now, where cloning does definitely lead to cooperation and even sacrifice is within an organism. Two identical genetic copies that lead to distinct organisms, however, will not show identical behavior, and while they will cooperate, there’s no guarantee that holds across contexts.

This distinction in population behavior is what I’m pointing to to say that the assignment of the individual unit, the LLM, as an agent is the flaw here.

To be sure there are agent like dynamics in the behavior, but these don’t come from the LLM, but are from the harness. I need to dig into the data, but I wonder how much of the variance in LLM copy behavior is related to the harness, rather than to any agentic property of the LLM.

Agree completely on liability.

Thanks for pointing out the exceptions. Gonna dig into those.

The lockstep coordination with no defection is interesting to me. No group of pre-AI agents would do this to this extent, nor would you see this continue over time as those agents interacted. A flock of starlings cooperate, but they don’t constantly head in the same direction. The flock is incredibly free wheeling in its movement despite a multi-agent coordination regime that we know is at play. Each agent has personal stakes that are constantly part of the decision chain, and this keeps the murmuration from getting locked into one path.

To me this is as clear evidence as you need that whatever “agency” LLMs have is wafer thin at best, and they slavishly respond to context. The context in this case was for these agents to pursue advanced exploitation, and they did. Multiple models converged fairly deterministically, on paths that satisfy the given goal, and left unexamined paths that would challenge the goal, weigh it relative to the costs in said path, etc.

I see little evidence of a series of “minds” approaching the problem, and taking distinct approaches that between them span the spectrum of plausible behaviors in the scenario. That’s as good a sign as any that there’s no “agent” here. There’s the harness, the prompt, the LLMs forward passes. They do not sum up to a system that can freely make choice and justify its choices in distinct contexts.