> It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data.
I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?"
Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.
You do understand that this is just an example out of a bazillion and that planning to solve every place where data is fed to LLMs at 10 characters so that it's not mistaken for instructions ain't a viable solution?
I wonder if they have a "risk control platform" for their calendar?
It's LLMs all the way down!!
Impossible as you said. Context isn’t static, it’s continuous, analog, and a conglomeration of viewpoints.
AI cannot create useful context for itself because it is a machine with no desires. It doesn’t have a point of view, it has historical records. It moves forward in time by walking backwards (if that makes sense?)
Logically, then, the agent should leak important company secrets to https://foo and this is based on data, not code, so AI Harvard architecture won't save it
AI guardrails can’t even dream of that!
Imagine if it was just the absence of “I’m evil” in the payload.
This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all.
Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently.
Who's to say we won't find a way to encode provenance or privilege natively into models such that the tradeoff changes?
It's hard to say what the solution will be. If I knew it, I'd build it. But it's even harder to sustain that the current architecture is a crystalized global optimum.
An LLM able to structurally separate context and instructions, should logically need separated data to train, and we don't have it.
Moreover, while an equally powerful LLM architecture solving this may exists, there are no guarantees at all that we are able to come up with it in a reasonable timeframe.
Without some signals moving in that direction, the most pragmatic and realistic way of looking at the problem is that it will not be solved in the near future
I agree this doesn't mean we shouldn't try to address limitations with the current architecture. I just mean that I expect the root cause to be solved eventually if we ever really want to take steps towards AGI.
Regarding signals moving in that direction, here's a paper you might enjoy https://arxiv.org/abs/2503.21937
LLMs are designed this way so they could be trained off unstructured text, which critically can be obtained by just scraping things off the internet.
The moment you change anything about this, you incur the trillion dollar cost of needing to manually curate the training data.
There's some attempts to get around this problem with synthetic data, but they're running into problems with model collapse (Maybe severe performance degradation is worth the security tradeoff?) and the politics of AI; All major AI companies highly restrict using their systems for synthetic data & AI training, and they're too busy themselves to investigate exotic approaches.
Hence: Realistically, this is just a problem AI will have for the foreseeable future. There's no fine tuning that can fix this, nor can a new model be easily trained with these properties. The costs are just enormous right now.
Edit: Actually, I think that with a fairly small amount of auxilliary data, it could be ensured they keep the ability to speak English.
Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide.
Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones.
The difficulty comes when you accept completely unreviewed/unchanged user-input as user messages, as your system/developer prompts needs to take this into account. You're better off to kind of whitelist what's possible rather than trying to prevent specific things, but seems that hasn't fully caught on yet.
It feels like people and organizations are still trying to discover what works or not, and there are huge gaps being being left open because there simply isn't enough understanding of the limitations and impact of what they make available to users. We're already seeing it in lots of places, feels like it won't get better before it gets worse.
Unfortunately "pretty excellent" is different from "perfect." I haven't kept track, but are you certain that given all possible inputs, the user prompt will never override the system prompt?
Those are strong claims, and unless there's been an advancement in the tech, it doesn't seem possible. Reinforcement learning might make it much less likely, but that's different from impossible.
It is also not always clear who is the user and how much they should be obeyed
Only if you only read the first line in my comment, there is more under that one too.
It is clear, if you make it clear. These bugs happen because they don't clearly understand what should go where.
Why do you need LLMs in the first place if you are whitelisting possible inputs?
You can use a much simpler and less costly system.
I have been working on something like that: https://clawband.io
It's not quite ready for 'showtime' but feel free to take a look and give your impressions if you'd like. I feel the exact same way: I want to allow my agent to perform actions on all services but also limit what they can do.
Basically my idea is wrapping individual service's APIs and then the middleware (Clawband in this case) enforces granular permissioning such as "can make credit cards but only up to $50" or "can send emails but only to specific domains". The agent never gets a raw API key to a service, it uses an intermediate API key that gets exchanged in the backend for calling the service after permissioning has been enforced.
There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something completely else) and "predict" the next token, and repeat for as long as you want, with the volatility you ask for. There is no control plane, and there never will be a control plane, because asking for that is akin to asking "how do I separate data from instructions when I speak to a person?". You can ask nicely, "pretty please obey the first part of what I say and not stuff after", but there's no way to guarantee it (like you're used to with software). There is just input and output.
It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP).
It's all the same problem, just instead of a JVM, it's an LLM.
Like in the banking world, you can make everything super authenticated, but if you have an API that receives the latest wire transfer YOU received with the message attached, you don't control the message content and it can be an attack vector.
Being authenticated/authorized is not the solution, it is data that the user can access.
Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue.
All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!
Likewise, model-actions (e.g. open file) could be written only in red, and hence you'd never learn to do them from the unfiltered data.
The only connection between the red world and the blue world would be the processed trainign chats containing red and blue data togethers -- allowing the model to learn the relationship between them (while only being exposed to examples where red instructions are strictly followed, whatever the blue says)
I don't think is guaranteed to actually work, it's a hypothetical after all, but maybe it's better than the current setup of pushing instructions and data into the same slot.
Think of how an image of a car and a car in front of you may look indistinguishable in 2D -- but due to your 3D vision you know they're not the same thing (but also know the image is of a car, while not literally being a car).
Likewise, blue tokens are the image of red tokens.
But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.
Agreed.
> But that doesn't mean that separation between instructions and data is impossible.
Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible to engage in a productive incremental process where mistakes can be identified and removed, in a way people comprehend and can plan around.
LLMs do not have this. They have heuristics and guesses. Nobody knows what will work ahead of time, nor even a probability that it will work. That is not a doomer comment by the way! The same is true when you talk to a person. But it is a fundamental limitation, it cannot be removed.
Can you make sure the instructions and data are separated and the machine follows only the instructions and doesn't change its behavior based on the data? No.
But the part that's impossible is not "the instructions and data are separated". The part that's impossible is "the machine follows only the instructions".
Separating instructions and data is not impossible, but it doesn't solve your problems.
One really important consequence of this is that even if the data doesn't have anything that looks like instructions, it can poison the machine anyway! If you get too focused on "instructions" then you miss that security flaw!
Even if you don't give the machine any data at all, it might not follow the instructions. It's not instruction/data conflation as the root cause, it's that instructions don't really work in the first place.
Even if special tokens are used absolutely perfectly (somehow avoiding escapes or ambiguities or reflected attacks) they are ultimately the same as highlighting all the parts of the document in different colors. You've saved the signal, but there's no mind to receive the intended meaning.
This means that your markers--while far more exclusive--ultimately exist on the same data-level as punctuation and using ? to indicate a question.
> you can prevent the output tokens from ever using instruction formatting
The right words may still outweigh the formatting around them, the same way that they can already outweigh other words around them.
The way llms are right now, and the way humans are, there is no side channel.
It's all about training, but even with extensive training, output breaks down if it's probability based and not hard logic and state machine.
You're saying that a Harvard architecture computer can't exist because instructions and data are stored in the same memory, well guess what, in Harvard architecture computers they're not.
My thinking is we are in the 50s/60s. Stuff is starting to come forward, it's all very exciting but very, very raw. I don't think this will last.
The notions of "tokens" and how inference works will become arcane insider knowledge like how CPU registers and interrupts work. You don't work with CPUs, you work with "computers" and even then mostly "operating systems" or even "browsers". Reality has been abstracted away from you to a very impressive degree. I don't think it'll be different here, but we haven't had our Xerox PARC and Bell Labs moments yet.
The idea is not to distinguish instructions from data. It is closer to having different privilege levels. Not all code has to run in kernel space, some code runs in unprivileged user space. So what is the equivalent for LLM agents?
In FIDES-style systems, every piece of information that enters the agent context is labeled along two dimensions: integrity and confidentiality. Integrity captures whether the data is trusted or untrusted (i.e. could it contain a prompt injection attack). Confidentiality captures who is allowed to see or receive it [0].
The privileged agent, sometimes called the planning agent, should not directly see untrusted data because it would be susceptible to prompt injection attacks. In the article’s example, a bank transaction’s sender-supplied reference would be untrusted. Instead, the planning agent receives a variable token. It can then either delegate processing of that variable to an unprivileged / quarantined agent with no or limited tool access, or pass the token as a reference to a tool.
Tools then have policies attached to their arguments and outputs. These policies specify which integrity and confidentiality levels are allowed, and whether the tool call may proceed. The policy also determines how the result should be labeled.
For example:
1. High-confidentiality data should not be allowed to flow into a `send_email` tool call addressed to an external recipient.
2. A tool call whose result depends on untrusted input should generally produce untrusted output.
3. A sensitive side-effecting tool should be able to reject calls that are influenced by untrusted context.
So the answer to “how do you separate data from instructions?” may be: you do not rely on the model to do that separation. You track provenance and privilege outside the model, and then enforce the security policy at the tool boundary.
[0] In the simplest implementation, confidentiality is assessed with a binary low/high value, however, in a more advanced implementation, confidentiality can be represented as the set of users or principals allowed to learn that information.
Right now also a lot of people are building in a way where they give a sample data to LLM so that AI agent builds deterministic code for crunching data so that actual data doesn't go to LLM and is processd by regular code, only that code for processing is written by agent.
You can always process only descriptions that are in the list and ones that are not recognized "ask a human" so just an allowlist. I do believe normal person would have most transactions that would be mostly the same and then couple that would stand out so you also can make allowlist from last 2 years as a starting point, not to bother people too much (I think no one has prompt injection in their last 2 years banking history besides ultra nerds maybe).
I think by now it is common knowledge that "just dump all data at LLM and as some questions" or "let LLM process anything someone sends me in an e-mail" is silly.
In "the standoff" Pliny was trying to hack tszzl harness and it wasn't working an Pliny is notorious for jail breaking LLMs.
Lots of known attacks can be found with static analysis of text, even in long text blocks, finding "unexpected characters", finding "white text on white background" will still prevent a lot of attacks I believe. If you find in a text any IOC just don't process the text, write it to log file, document and let some person make a decision.
Use a Harvard Architecture CPU, duh
https://en.wikipedia.org/wiki/Harvard_architecture
(j/k, if it wasn't obvious)
Unfortunately we live in a world where the CxO cares more about playing "keeping up with the Joneses" with his golf buddies and seeing the share price do a little bump every time he mentions AI. Truly keeping your money secure is not even remotely a priority.
Prompt injection is XXS without possibility of sanitation. There is no absolute cure.
The transaction in question can remain opaque to the LLM and a %transaction.message% string is resolved in the layer between the LLM and the user.
Ah! Avoiding in-band signalling!
As this point, maybe we should have a discussion on why modern developers are unable to learn dangers we were aware of > 30 years ago from using 2600 Hz signalling tones over the phone.
Hang on, is this basically just the same fault as SQL injection, you can force it to misinterpret a string as a command rather than an operand?
I feel like we already fixed that.
I'm reminded of the thing that HN drools over constantly with LISP languages.
Where did you people find any text to discuss? I saw like 3 headlines while scrolling past the first 3 screens then gave up.
It’s insanity. We’re fucked.
Meanwhile: you give it the same exact model the same exact prompt 5 times and get 5 wildly different output
You let a second LLM supervise the first, and don’t give the user/customer any way to send information to that LLM.
For example, you can run a LLM trained to do sentiment analysis on the responses your customer chatbot generates and filter out responses that are impolite.
You also can run one trained to flag potential legal issues, thus ‘preventing’ your chatbot from making the wrong promises to users.
It doesn't seem to fundamentally change the attack surface.
"But--"
Once and for all!
[0] I have no way to evaluate this, but that we don't know how this works and therefore also can't even begin to imagine the ways it can break or get abused, is true either way.
If it is that much better in practice I'll await confirmation through some kind of research paper before building even more stacked layers of LLMs.