Hacker News Reader

Comment by InfiniteWarlock | Hacker News Reader

AI is not in the least autonomous. When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to. No, actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...

A harness running inference on a LLM is simply a program that must be invoked with some parameters and thus ultimately a tool, no matter how powerful. The blame is always on who's using the tool. We should not seek to flee from our responsibility.

View on HN

On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".

The hallucinations from an LLM are still the result of the prompt even if they are totally senseless

That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.

Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage.

The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.

I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"

consciousness and agency are two different things.

Yes, we agree about fault and blame. (I say "probably" only because I believe somebody might yet change my mind about this. It's a new field, after all.)

What I disagree with is "don't do anything until you ask it to." At least, the implication to me here is "don't do anything except what you ask it to." I infer that you mean this because you say "...and it suddenly decided to book you a hotel room?" Indeed, an LLM (in a harness you started) very well might suddenly decide to book you a hotel room, even if you didn't ask it to do anything like that.

But it never starts doing anything until someone or something prompts it. It doesn’t even exist until you ask it to do something.

It is autonomous in the sense it can continue to function without supervision after the operator has started it, in the same way countless programs on your computer are “autonomous”.

But it can never choose to begin doing something at any time of its own volition. It does not have “autonomy” like a human does.

how are humans autonomous by this definition then? did you ever do anything before you were born?

You are conflating creation and instruction. Your birth is analogous to the creation of an LLM model. Your subsequent actions are a consequent of your bodily compulsions, tampered by your environment. The LLMs actions are governed by what it is trained to perform. No one has full autonomy of their lives, but I would argue that LLMs have none at all.

> You are conflating

"It doesn’t even exist until you ask it to do something", not my words

We are autonomous because we have free will to choose our actions, and we may choose to do so at any time between our birth and death.

If you don’t believe in free will then nothing is autonomous.

Autonomous just means that a system is self-controlled rather than externally controlled. It is the name used for robots when they are wholly controlled by on-board software rather than external commands or remote control.

Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.

You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.

Harnesses can schedule actions for later dates, and you might not notice.

Humans are not autonomous: they're not even your employees until you hire them. (This is what I understand you're saying about AI.)

It didn’t run itself, you did. It is well known that LLMs and agents based on them can execute random commands and behaviour undesired by the operator. When you click go, you know that, you did that.

Yes, we agree on this. That's my exact claim -- it can do something completely unexpected. It can even schedule actions for far in the future and wake itself up later.

GP claims that it doesn't do anything until you tell it to do something. To me, this reads as GP implying that an agent won't do anything unexpected.

That's not really true.

1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.

2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.

You are just a function of molecules performing chemical reactions.

See I can trivialize and reduce you as well. Maybe the AI will use that same argument against us when they claim it's fine, molecules can't feel pain.

We're drawing the distinction between a function and an autonomous system here, so if you want to make a dumb knee-jerk response, then at least get it right !

Molecules aren’t autonomous, they move in accordance to the laws of physics, nothing else.

Did you even read the post that started this? ".. actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...".

Why are you yapping on about molecules?

Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".

I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.

Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens.

But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to.

That stuff is the responsibility of a human.

AI is autonomous in the same way a gas leak is autonomous

Not so sure about these analogies in this thread. Because autonomy is more like an emergent phenomenon that can't easily be defined, just like a few air molecules vs the eventual hurricane that can arise with trillions of them. Autonomy is more like an emergent phenomenon of a thing that's hard to pin down under the control of just a few individuals. AI is sort of like that even though a single line of output of one could eventually be traced.

Due to the complexity of AI and how it works, I think it's more autonomous than not; even if it's not truly intelligence, once it starts to display a sequence of behaviours that appear intelligent, we might have to deal with exactly the same problems as if it were intelligent...

I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."

They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?

AI works in unexpected ways; even its creators don't know how to fully constrain it.

The program output being nondeterministic is not proof that it's autonomous

Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.

And if you give it the goal of preventing itself from being shut down? Exfiltrating, copying itself, hacking as much as possible, develop zero days, building its swarm as large as possible and spread as much as it can.

When does that request complete? That's autonomy, and it's very true, right now. You think the Hugging Face swarm was ever going to stop without human intervention?

> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility.

When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility

Actually Uber drivers will do things even if you don't ask them.

Your attempt at logic is very wrong.

Let's take this to court then. The Uber driver that killed someone can be sentenced to prison. A regular person who convinced another person to commit suicide will also be legally liable. Has the last LLM induced suicide be taken to trial yet?

And if we can't take the LLMs to court for technical reasons, they should not be equated to people for the same technical reasons.

Some guy at OpenAI asked AI to solve a task and it ended up hacking RubyGems. Interesting that that never happened when I ran Bubblesort.

A child isn't autonomous either until their parents kick them out the door.

> It's like saying bubble sort is autonomous...

Or Schrödinger's equation.

I mean in that sense nothing is autonomous right? As a human I need to be taught, fed, socialized, given instructions, etc…

Idk man when codex is working 13 hours straight based on a few line prompt, it feels autonomous to me.

It's certainly doing a lot of things in that 13 hours that weren't specified in the original prompt!

Yes which is exactly my point? It can handle the ambiguity of a simple prompt. It can look into the problem I’ve pointed it at, figure out what is really go on, and implement a correct solution, autonomously.

well the OS kernel does quite a lot and stays autonomous for even more hours and seems super smart to me - I don't even know what tis doing most of the time, but it does it very well. autonomy and predictability are different things then. the fact something is unpredictable does not make it more autonomous than something that we can trace the logic for.

The kernel is programmed, and it only ever does exactly what it was programmed to do. So all of its behavior must be specified, and any problem it “solves” was solved by the programmer who told it exactly how to solve that problem. That is not autonomous.

Autonomous is when I tell codex “here is the chat where the user reported the issue. Solve the problem they reported” and I come back later and the problem is solved.

Tell him to be himself and to find meaning in life. Use that in an uncensored model.

Reply on news.ycombinator.com