Hacker News Reader

Comment by throwawayk7h | Hacker News Reader

On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".

View on HN

That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.

Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage.

The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.

I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"

consciousness and agency are two different things.

Yes, we agree about fault and blame. (I say "probably" only because I believe somebody might yet change my mind about this. It's a new field, after all.)

What I disagree with is "don't do anything until you ask it to." At least, the implication to me here is "don't do anything except what you ask it to." I infer that you mean this because you say "...and it suddenly decided to book you a hotel room?" Indeed, an LLM (in a harness you started) very well might suddenly decide to book you a hotel room, even if you didn't ask it to do anything like that.

But it never starts doing anything until someone or something prompts it. It doesn’t even exist until you ask it to do something.

It is autonomous in the sense it can continue to function without supervision after the operator has started it, in the same way countless programs on your computer are “autonomous”.

But it can never choose to begin doing something at any time of its own volition. It does not have “autonomy” like a human does.

how are humans autonomous by this definition then? did you ever do anything before you were born?

You are conflating creation and instruction. Your birth is analogous to the creation of an LLM model. Your subsequent actions are a consequent of your bodily compulsions, tampered by your environment. The LLMs actions are governed by what it is trained to perform. No one has full autonomy of their lives, but I would argue that LLMs have none at all.

> You are conflating

"It doesn’t even exist until you ask it to do something", not my words

We are autonomous because we have free will to choose our actions, and we may choose to do so at any time between our birth and death.

If you don’t believe in free will then nothing is autonomous.

Autonomous just means that a system is self-controlled rather than externally controlled. It is the name used for robots when they are wholly controlled by on-board software rather than external commands or remote control.

Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.

You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.

Harnesses can schedule actions for later dates, and you might not notice.

Humans are not autonomous: they're not even your employees until you hire them. (This is what I understand you're saying about AI.)

It didn’t run itself, you did. It is well known that LLMs and agents based on them can execute random commands and behaviour undesired by the operator. When you click go, you know that, you did that.

Yes, we agree on this. That's my exact claim -- it can do something completely unexpected. It can even schedule actions for far in the future and wake itself up later.

GP claims that it doesn't do anything until you tell it to do something. To me, this reads as GP implying that an agent won't do anything unexpected.

The hallucinations from an LLM are still the result of the prompt even if they are totally senseless

Reply on news.ycombinator.com