Hacker News Reader

Comment by simonw | Hacker News Reader

My favorite line from this paper:

> The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment.

This is the key thing people need to understand about why prompt injection is such a critical issue, especially now everyone is wiring LLMs together with tools and MCP servers and building "agents".

View on HN

This reminds me of the Perl concept of taint. Once an agent touches tainted input, it becomes tainted as well (as you mention in the article), the same in Perl (when you operate on tainted data, the result becomes tainted).

Yeah, I think taint tracking was one of the early ideas here also.

The problems is that the chat context typically is immediately tainted as for the AI to do something useful it needs to operate on untrained data.

I wonder if maybe there could be tags mimicking data classification - to enable more fine grained decision making and human in the loop prompts.

Still a lot of unknowns and a lot more research needed.

For instance with Google Gemini I observed last year that certain sensitive tools can only be invoked in the first conversation turn / or until untrusted data is brought into the chat context. Then for the next conversation turn these sensitive tools are disabled.

I thought that was a neat idea. It can be bypassed with what I called "delayed tool invocation" and usage of a trigger action, but it becomes a lot more difficult to exploit.

It seems to me that the only robust solution has to be some sort of split-brain dual model where tainted data can only ever be input to a model which is only trained for sentence completion, not instruction-tuned.

Untainted data is the only data that can be input into the instruction-tuned half of the dual model.

In an architecture like this, any attempt to prompt inject would just find their injection harmlessly sentence-completed rather than turned into instructions and used to override other prompt instructions.

Yeah, improving robustness from prompt injection which such techniques will help.

One attack avenue that is surprisingly not discussed much is that the model itself can be the attacker.

In that case prompt injection is not the root cause, but a misaligned/backdoored model that might invoke tools is.

So super risky use-cases should always require human oversight, but I'm worried we are already on a path of normalization of deviance.

It's sort of the unlikely worse case scenario, but Murphys law reminds us that such an attack/accident will happen one day.

https://perldoc.perl.org/perlsec#Taint-mode

One of the tricky things is untrusted input somehow making its way into what is otherwise considered trusted input. There are obviously untrusted inputs like a customer support chatbot. And there are maybe trusted inputs, like a codebase that probably doesn't contain harmful instructions, but there's always a chance that harmful instructions might be able to make their way into it.

Very helpful Simon! I have definitely been hesitant to spin up any agents even in sandboxes that have access to potentially destructive tools, but this guiding principle does ease my concerns a bit.

Reply on news.ycombinator.com