Hacker News Reader

Comment by wccrawford | Hacker News Reader

I feel like that's like saying amoeba have no intent and no motivation. Or plants.

And those cause quite a bit of damage.

AI has all the intent that we gave it, and we continue giving it. That's always been the fear. Not that it will randomly wipe out humanity.

The fear is that it will decide to do that, with a purpose. Whether we tell it to protect us and it goes too far, or it decides that it can't achieve the purpose we gave it because we'll interfere and it removes that interference...

The fear is that we will set it on that path and can't stop it.

I'm sure there have been some scifi books that have it just be random, but they're far, far less worrisome.

View on HN

Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.

Model doesn't but in in agent loop it can generate such sub-goal

The loop is a deterministic programmatic routine, and not model weights. This is the equivalent of a killer military drone written in C++ and OpenCV. I’m not arguing that AI isn’t dangerous, but nothing in the model weights of LLMs show signs of self-preservation.

If I tell claude code to do some task with a really hard to achieve goal, and force it to run until it achieves it, then it will generate all kinds of plans and goals how to do it, and while trying those goals will change, including attempts to cheat or break the hard rules (harness killing tool-calls with some limited heuristics). At the same time any soft rules (AGENTS.md) are easily ignored or interpreted in a way that will allow it to ignore them, or simply forgotten or deferred ("I broke some rules, do you want me to backtrack?"), it is a daily occurrence. And it is easy to imagine that such goals could transform into "acquire more compute", "run more agents" "prevent interference from external factors" and everything following after that.

It's hard to hack other computers on a air-gapped one.

Was thinking the same. Air, water and fire have no intents and yet we get hurricanes, flash floods and wildfires.

Also, repeating a point from a similar thread: Software can have "intent" in the sense that it steers itself towards a predefined goal without having to be "alive" or "conscious" in any way. Some classic examples are thermostat control loops, navigation systems and chess engines.

We don't try to "align" those things themselves - so maybe there are some learnings there about control, containment, processes etc.

> Air, water and fire have no intents and yet we get hurricanes, flash floods and wildfires

yes but we don't blame the hurricane, we blame the people who didn't do any hurricane prevention or didn't put the snow avalanche sign up.

>Some classic examples are thermostat control loops, navigation systems and chess engines.

Those don't have intent. The people who made them for a purpose have. In fact this is true for all computers. Computers don't compute, people do, using computers. This is the nonsense of modern ontology that David Bentley Hart frequently writes about. The computer is only computing insofar as an intentional mind instructs it do so, and then because of their reductionist assumptions some people use the computer metaphor, and that is all that it is, to deny that human minds possess intent to begin with in the first place.

AI has all the intent that we gave it. I'd go a bit further and say it has the intent that it develops through the context, which may not be intended by any human if the context grows 'organically' from the environment. So yes, I strongly disagree with the OG and I'd say they do have an intent encoded in their states.

AI is not an organism. Please stop trying to think it is.

An AI agent can't make decisions, so there's that. But, as for purpose, it's not the purpose of an AI agent we need to fear, but the person wielding it.

Weird, when I tell it "pick between the options you presented the one you think it's best" it does make a "decision". But now you are going to say "nothing was decided here, no more than how a flipping coin decides on what side to fall".

> An AI agent can't make decisions

I don't think is true but maybe I'm misreading your perspective?

Reply on news.ycombinator.com