Comment by themgt | Hacker News Reader

Comment by themgt | Hacker News Reader

The fundamental point I think is far too often confused is the difference between LLM and agentic system.

An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!

Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.

Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:

1) LLM - can't take over german wiki because it just generates tokens

2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory

None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.

If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...

"The coin don't have no say. It's just you."

With respect, I think the distinction is irrelevant to the actual point. Whether the definitions are accurate enough means little to the actual problem, which is that we are rushing towards creating extremely powerful tools. Recorded (and arguably unrecorded) history seems to demonstrate the same pattern: inevitably, any tool will be put to use towards violence. I would argue that yes, it’s a tool and humans are the trigger, the same way that “guns don’t shoot people, people shoot people” is technically correct.

The real question then is: are human beings responsible enough, en masse, to wield these tools without restrictions? Would you let a toddler play with a loaded gun, even though without human action the gun is harmless? Maybe collectively we’re all little better, and eventually one of us is going to “pull the trigger” of ai just because it was shiny.

Ironically, I also don’t believe in regulation, unless that regulation is crushing and draconian. The only regulation that would seem to make sense is status quo anti ai, with only highly supervised research labs being able to interact with it. Clearly that’s unrealistic without the preceding calamity: millions had to die in WWI before we instilled a global taboo against chem weapons on ourselves. I don’t want either to happen (regulation or calamity), but I feel individually powerless to stop the relentless drive we (and I) seem to have in curiositying ourselves to death. So I’m stuck with all the rest of us, building SaaS apps and iPhone games.

No, we mostly stopped using chemical weapons because they were bad at modern warfare :

https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...

Maybe true, but also:

https://en.wikipedia.org/wiki/Geneva_Protocol

https://en.wikipedia.org/wiki/Chemical_Weapons_Convention

https://www.jstor.org/stable/2706867

https://www.foreignaffairs.com/articles/syria/2013-01-22/how...

Well, that's kind of like saying that brains can't do anything other than trigger weights on neurons.

They're part of a whole system.

The bad part is that it does not require a collective decision. It just takes a few key people.

Some OpenAI researchers neglected their sandbox safety for a few weeks/months and thus hacked HuggingFace. Maybe eventually that is sufficient for the AI to secretly buy its own compute and keep running there even if the researchers shut it down in their lab.

I’m not sure what the point of this distinction is now because agents are where all the AI is now. And they’re just going to be given more and more tools and be allowed to make their own tools - or misuse things they find and turn into tools, or make them anyway without anyone noticing.

I don't remember where I saw it but there was a woman in one conference who very eloquently put it that since frontier AI companies took humanity's work to train their models [without explicit permission of every single person who contributed], then those models belong to all of us and should be open. I think it is a very grounded perspective.

And I agree that LLMs themselves are harmless, it's the people behind them driving the agents and instructing them to do bad actions are the ones posing harm. But if the models were open and everybody could operate them on equal grounds, then the good guys would have the means to fight back against the bad guys, in a very simplified sense, just like they did before the LLMs. Everything stays balanced, just both sides get faster.

I agree with this. LLMs can actuate over capabilities we expose, so the blame isn't only on LLMs for this.

This feels like a distinction without a difference, like the endless wrangling over which piece of metal in a gun legally constitutes a firearm. The combination of the two may or may not be dangerous, but it's definitely the more useful combination, so of course that's how it's going to be set up.

This is a pointless essay. The point is the person in the article believes systems can cause harm to humans. How we label those systems is irrelevant.

You write as if you are a lawyer for an AI corp trying to avoid a judgement. It’s akin to saying Teslas FSD/Autopilot can’t kill anyone, it’s just a computer. Cars can kill people. Totally different things. FSD/Autopilot is safe by definition and no one should try to legislate it.

“a prompt telling it to attack stuff” — that’s the uncontrolled or not fully understood part of the AI isn’t it? No human told the deterministic system in 2) to attack Hugging Face. The random token generator landed on a guidance for 2) that caused it while 2) in itself still remains a deterministic system.

There’s just also so much confusion about LLMs and Agents because they use language outputs and can do these higher level reasoning tasks, ie given an objective can programmatically decide their own course of action, like the ultimate self-writing programs. Then comes “well what if we just take the guardrails off and let it go bananas and call it superintelligence”.

Before we get too gung ho with this though, it needs to remain a focus that any programme in the world producing strange or not well understood outputs and behaviours is not something that needs license to run rampant in the wild - it’s not intelligence; its not some mystery magic box that does amazing things; it’s code, same as all the rest. Letting that code run without oversight is poor human behaviour same as all the rest and that’s where the irresponsibility lies, likely triggered by forms of AI psychosis and tunnel vision on the part of those running companies like Anthropic and OpenAI.

It’s irresponsible behaviour to give a chimp a machete and set it loose in a shopping mall. It’s irresponsible behaviour to let an elephant in heat freely wander round a circus. It’s irresponsible to fire off nukes the first time we’re under threat… so none of these things are ever done. Yet this is what we’re proposing to do with these hyper-complex GPU algorithms while building massive server farms burning power-stations of energy to ensure they can run freely - it’s harebrained, irrational behaviour and for all our stupidity we are usually good at letting sense prevail towards these things once the bells and whistles wear off or a couple of coolies get stamped on.

So you think the a lead researcher at Anthropic is confusing LLM's and agentic systems? That's not really a conclusion you should come to.

Reply on news.ycombinator.com