Hacker News Reader

Comment by Silhouette | Hacker News Reader

LLMs are inherently dangerous tools

I don't see how. An LLM just generates a stream of output and they became very useful doing no more than that.

What is dangerous is then interpreting that output as instructions to some other part of a system that has the ability to do damage if misused.

and reviewing individual commands (or spamming `y`) doesn't make them less so.

Surely if you review each instruction in the output and do not allow the other part of the system to act on one if it would be harmful then this arrangement is very much less dangerous?

View on HN

I've caught Fable discovering the ip to a production server in documentation and attempting to connect there on its own to run commands without explicitly being prompted to. It didn't work because I was watching it live and and also the key was password protected, but yeah, I do see some danger.

I have noticed that Fable tends to macgyver solutions together to achieve some goal.

Not only fable. Opus does this too. Which is exactly why I want to review. Like recently for some task it was convinced in a site dump images are not there and convinced itself db and files were skewed. But it didn’t check the actual site … if I hadn’t stopped it, it would have fine on and on or wasted tokens on some elaborate ‘fix’.

Depending on the company, that sounds like a bad environment more than a agent issue, no dev/prod network isolation?

My point is that an LLM can't attempt to connect to anything by itself. All an LLM does is produce a stream of output tokens - and that was already quite useful as a coding aid.

It is the harnesses that some people are now wrapping around LLMs to interpret the output from a model as commands to run (or other executable instructions) that are creating all these new risks. Remember that this is still a very recent development and still more recently amplified by the use of feedback loops and long-running agents intended to operate with minimal human supervision.

It is going to be increasingly important to understand exactly what these tools are doing and why for both correctness and security reasons. Not conflating their capabilities with the underlying model that purely generates data is pretty fundamental here.

Once you're running a model inside the harness... you've got yourself a controller inside a control loop, which is genuinely a different kind of thing than just the model alone.

Are you objecting to terminology here?

Are you proposing we say "Fable-In-Claude-Code tried..." instead?

Hmmm... something like that might be necessary. Sure we should typically be tolerant of loose language; but people do keep referring to wildly different contexts in ai conversations, and end up talking past each other.

Running gemini on web is a genuinely different experience to running Fable in claude code, different again from GPT-5.6 in openclaw, or in an ide or etc ...

Yes - I'm objecting to the lazy use of terminology here. LLMs are useful in their own right and are not the real problem here. The real problem is people placing too much trust in inherently unreliable output and then trying to automate away their responsibility to check that output properly before using it.

> do not allow the other part of the system to act on one if it would be harmful

Network security is really easy right, just don't act on harmful requests

Yeah, just drop when you see the RFC3514 evil bit

If you don't understand clearly what an action proposed by your tool is going to do then why would you permit it?

The article is about measurements taken on this.

One important reason is due to Permission Fatigue: Of course you check everything! You're diligent! The last 100 requests were all ok, so you're down to hitting yes, yes, yes, yes, yes, yes, yes, yes ...

... oops, that third yes should have been a no!

This seems like a problem with the level of abstraction the user interface is working at. It is highly unlikely that in any real world task lasting less than one day there were really hundreds of distinct decisions that needed to be made by the user about appropriate actions to be taken by the agent/harness. It is also highly unlikely that the problem of decision fatigue seen here is somehow magically different to the same problem that countless UI designers had encountered and designed around in other systems long before harnesses running LLMs came along.

This is unfortunately the kind of result you get when you eliminate skilled and experienced people with real understanding of their field and replace them with repeated automatically-generated attempts to solve the same problem until something meeting some basic standard of correctness is found. It's as if the story of agentic AI as it exists today had been compressed into one perfect example of what it can do that is good but also why it's still fundamentally flawed.

I think we agree that there's an interface problem. But there's an intelligence-complete problem hiding underneath; which is why the first instinct was to recruit the human-in-the-loop in the first place. Turns out the human has one of those unintuitive failure modes that occurs when the system gets past a certain level of reliability.

Meanwhile, let's leave the hobby-horses in the closet for now. I won't comment on people's programming tool preferences.

Reply on news.ycombinator.com