It feels like the models have been very optimized at getting shit done. But not so much at figuring out what the limits should be.
That is still dangerous and it shows that the models ARE misaligned with what their users are wanting/asking them to do.
No, but the LLM didn’t decide anything. It followed the prompt. That’s all LLMs do.
This whole thing is like playing russian roulette then getting mad at the revolver.
If you wire /dev/rand up to a bash shell you don’t get to be surprised when it rm -rf’s your machine.
For instance, I was playing around with Claude a few days ago and it decided that it was missing a tool and it was going to get it one way or another.
First, it tried apt. No sudo, so no install that way. Tried installing via mise, but it didn't have the permissions. Then moved on to grabbing the source from github and building it.
Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
> Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer.
> Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
> If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
We've drifted onto architectural issues here but I will say the only way to properly limit these things is to apply actual hard constraints.
I think the typical pattern of giving them a bash prompt and a filesystem to play with is foolish, and has far too many gaps. My preferred technique - when I have built 'agentic' systems (e.g. years ago I built a small MUD with LLMs pretending to be NPCs) - is to allow them access to a customized lua interpreter embedded in the harness and nothing else. Then, you stub out lua functions for allowed actions like web searching, math, etc.
The lua sandbox then provides isolation and a clear layer for access control mechanisms. When it tries to make a network request, you pause the whole thing and wait for human approval. No trying sudo, no installing stuff, no trying to compile stuff, it gets to call lua functions and output text. Which are the same thing really.
> The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
> But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
Unfortunately, LLMs won't ever reliably 'notice' and comply with such things because an LLM is essentially a complicated constraint solver. They are over-optimized on problem solving, but that is a natural consequence of the way they are trained. They aren't living, thinking beings, and so they aren't trained in simulated environments - they're trained to output the "best" response for a given prompt and then emit a stop token.
They are, essentially, like a ball rolling down a hill and they will take the easiest path forward at any given point.