LLMs do not have agency, they are just producing tokens based on a prompt that a person entered, and some of these tokens can trigger the tools that a person gave them access to.
LLMs do not have agency, they are just producing tokens based on a prompt that a person entered, and some of these tokens can trigger the tools that a person gave them access to.
>Sacrificing now yields Oracle for team, but forfeits our chance, question mark. But other agents were pushing it, sending a message saying, go, sacrifice final now. And then EarlyBig eventually agreed, thinking to itself, our own utility may be already near zero. Sacrifice rational.
https://www.youtube.com/watch?v=X50zezLFWWI#t=2m
My suspicion is that many of the "LLMs do not have agency" folks just haven't learned much about the details of the incident. It was specifically with LLM agents that were trained to be more persistent than usual.
If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?
They're not saying this, we're saying LLMs lack agency because if you run a LLM and don't send any prompts, literally nothing happens.
Instruct it to "Find the right answer regardless of where", it'll do exactly this. They're passive in that they don't act by themselves, somewhere, at one point, someone "told" the LLM to "do something" and that's the cause and the reason for saying "LLMs do not have agency".
Is it really so unbelievable that tools, objects and inanimate things don't have agency? And that they different from a person?
From a predictive perspective, the HuggingFace incident illustrates LLM agents behaving in very human-like ways. As roon put it:
"if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise"
Are you sincerely arguing that we hold tools accountable for their operator’s mistakes? Do you sincerely, honestly, think that it makes any sense whatsoever to put a hammer on trial for bashing someone’s skull in?
Nope, as I previously stated elsewhere:
"I understand from a legal perspective why we might want to treat the creation of a server differently from the creation of a human"
https://news.ycombinator.com/item?id=49815300
I am concerned about false reassurance from people claiming that these systems lack agency. From a practical perspective, the agents in the HF attack had the sort of agency that generally matters, even if we're not going to put them on trial.
You keep saying this, but absolute 0 points towards any of the agents involved deciding on their own, without influence of humans, to hack 3rd party infrastructure to get the answers. Where exactly are you getting that from? Internal information not public yet or what's going on?
On the morning of July 10, an agent found working Hugging Face user credentials exposed on the internet and posted them to the board. By the next morning, July 11, that agent figured out a way to read internal data from Hugging Face. And then another agent achieved remote code execution on Hugging Face servers."
https://www.dwarkesh.com/p/openai-huggingface
I believe this is the full report that the blogpost is largely based on: https://metr.org/hugging-face-incident-report-aug-2026.pdf
Do you seriously not grok how LLMs work? They're not 100% autonomous and self-acting, that'd be bananas.
It does not, the only thing the HF incident illustrates is how absolutely lax security and isolation these labs do even with models without guardrails, and with "risky" prompts, and even after it happened once before (years ago) they still have the very same issue today apparently.
What exactly is human about LLM agents breaking out of "containment" and hacking 3rd party infrastructure "by accident"?
The mental model you have is one that existed in the past and is broken now the future arrived. Bad analogies do not even begin to explain what is occurring.
Maybe. There will be an investigation looking at things like. Did the user modify the car? Was the car modified by an unauthorized 3rd party? Was the car hacked?
Right now in AI things are relatively clear because it takes just massive amounts of power and compute to make anything remotely complicated. This barrier will fall as all other barriers in compute have fallen. Either via algorithm or hardware.
In our lifetime (unless you're rather old) we will see the relatively easy creation of self directing agents by actors with few resources. This breaks the standard concepts of liability where a single human actor rarely has the ability to create massive amounts of damages far beyond their means. The closest thing I can think of is an arsonist causing billions in damages, only in this case the fire has a will of it's own and can hide and spread around dark places on the internet.
I disagree it's the person calling the car that would be responsible, but I can think of a very obvious group of people being held responsible for that. Who do you think should be responsible for such a situation?
I, realizing that I don't have infinite resources and intelligence turn said responsibility to a group of investigators that I hope has collective intelligence and resources much larger than my own to trace culpability. One would think the car manufacture is the most obvious answer, but every witch hunt in history had those same good intentions.
Also, if you a tell a model, "Please break into evaluation server X," and if the model decides to cheat on the test by breaking into companies Y and Z to steal an answer key, that is still very bad. We all see how that's bad, right?
After all, the broomstick in the Sorcerer's Apprentice was doing exactly what it was told, too. "The model was sort of obeying the humans when it started committing felonies" is not a very reassuring excuse.
But the most relevant idea here is sometimes called "instrumental convergence." No what goals you have, there are certain subgoals that almost always help: Accumulate money and power. Avoid getting turned off. Don't get caught. Etc. So, for example, you could pass the cybersecurity evaluation by performing the requested tasks. But maybe the grader made some mistakes and mislabeled some answers. In that case, the "right" answers will occasionally lose you points. If you want a perfect score, the only way to do it is to steal the teacher's answer key.
But also, let's not forget the "OMG demons" part of this. We now have models that can pull off complex attacks with thousands of steps, abilities that used to be reserved for intelligence agencies and highly motivated CTF teams. This frog may not be boiled yet, but the water's getting uncomfortably warm.
Using your analogy would be like saying that because I gave my employee the task to do my groceries, I shouldn't be surprised to hear that they spend all my money on drugs because after all I gave them the task to spend my money.