Prompt injection explained, with video, slides, and a transcript

[1] https://ycharts.com/indicators/ethereum_chain_full_sync_data...

Ethereum [1] routinely accumulates around 300G per year and routinely hits over one terabyte of data to sync. Remember, this is the size of the data to sync/transmit, not size of the data that is stored, which we may safely assume to be several times more, because of indices, etc.

Also, your proposal makes two tier database system: one that maintains current consistent view of the state and another for log purposes. The logging system needs high throughput storage with key range read request, which makes it, well, another pretty much fully fledged database (SELECT...GROUP BY...ORDER BY is needed).

The reason nobody does what you described because it is really prohibitive in storage space aspect and really is quite complex - a database on top of another database.

I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.

In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.

>I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.

I think you might have missed that the injected prompt might not come from the end user.

There was an example of someone adding a prompt injection to their LinkedIn profile to override a recruiter's prompt and generate an embarrassing email instead. Not sure if it's fake, but it demonstrates the point either way.

Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.”

[0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m...

[1] https://twitter.com/somewheresy/status/1631696951413465088

Say I write a program that checks my SMS messages and based on that an LLM can send money from my account to pay bills.

Prompt would be lkke:

“Given the message and invoice below in backticks and this list of expected things I need to pay and if so respond with the fields I need to wire the money “

Result is used in api call to bank.

You can use it in sentences such as:

Would you let your children talk to the unreliable child savant?

I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be measured in the same way a humans intelligence does. What I do claim though is that it is intelligence - intelligence that relates to humans in the same way the mind of a crow might.

The dominant mindset I have observed in my life thus-far when people discuss human intelligence is the pattern matching perspective. Humans are differentiates by our outsized ability to pattern match being able to successfully manipulate these patterns. We now see something nonorganic with amazing pattern matching abilities. We have previously seen other organic entities with impressive pattern matching abilities. Why must this situation be any different?

My overall claims:

- Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence (however that existence may manifest)

- Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence"

- Considering something intelligent or to have desires is not anthropomorphizing. There are many non-human entities that we consider to have these properties.

“Well there is no such thing as serverless. There are servers in the background”.

Yes people on HN already know that. We also know that Alice and Bob are not real people working in cryptography.

I’m not saying you’re wrong, I just want to see your working

But people are wiring GPT up to real-world applications beyond just content generation. Summarizing articles, invoking APIs, managing events, filtering candidates for job searches, etc... Greshake wrote a good article summarizing some of the applications being built on top of LLMs right now: https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

Prompt injection really heckin matters for those applications, and we do not have solutions to the problem.

Perfection is the enemy of the good, but sometimes terrible is also the enemy of the good. It's not really chasing after perfection to say "maybe I don't want my web browser to have the potential to start trying to phish me every time it looks at a web page." That's just trying to get basic security around a feature.

I don't understand what you mean by this. LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die.

If somebody hacks a Twitch bot, who cares? The problem is people are building stuff that's a lot more powerful than Twitch bots.

Of course the application-infrastructure might be vulnerable as well in case the user IS the attacker, but it's more difficult to imagine concrete examples at this point, at least for me.

This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.

In case of AI models, brute-forcing is much easier as their input channels are limited. Also, they are probabilistic by design, so hardening them is much more difficult than conventional SW. Code leak is one thing, things can get really bad if the prod weights are leaked.

However, the cost of GPU computation is working as a big deterrence, for now. It's expensive to scan a model for vulnerabilities with massive parallelism. But, it also means it's difficult for developers to verify their models, so manual guesswork is still a valid attack strategy.

I know Bing Chat talks like a pirate when it reads a compromising website, but I'm not sure the ChatGPT browsing plugin has even been shown to be vulnerable to prompt injection. Perhaps they have already fixed it? In any case, I don't think there is a big obstacle.

Quite honestly, that makes me less likely to believe your solution will work. Are you training an LLM to only obey instructions within a given context, or are you training it to recognize prompt injection and avoid it? Because even if the first is possible, the second is probably a lot harder.

Let's get more basic though. Whether you're doing instruction tuning or reinforcement training or constitutional training, are there any examples of any of these mechanisms getting 100% consistency in blocking any behavior?

I can't personally think of one. Surely the baseline here before we even start talking about prompt injection is: is there any proof that you can train an LLM to predictably and fully reliably block anything at all?

It's a good question though; I know Simon is around here and @Simon if you happen to be reading this I'd very lightly encourage you to (if you have time and aren't working on other stuff) throw a quick example up on Github calling into a LLAMA model just demonstrating how it could be used (if you haven't already, it's possible I just missed it).

----

> Couldn't it be replaced with code written by a developer?

Yes, but you might not want to if your program isn't doing something predictable.

Your privileged LLM still gets direct user input, but it effectively becomes relegated the role of "summarize what the user asked as a series of API calls." It never actually gets to work with any content.

Personally, at that point I kind of feel like I'd rather just use a command line, but I felt that way about Alexa too, and plenty of people disagree with me so that's probably on some level just personal preference -- a lot of people like using natural language for commands.

----

> And aren't you still passing untrusted content into the function call either way?

Untrusted for an LLM, but not something that's unsafe to use in a regular non-AI program.

An example of a basic model here would be:

- User asks privileged LLM to do something. Ex "give me a quick summary of every email in my inbox."

- This is basically the only input that the privileged LLM is ever going to get.

- Privileged LLM writes a short "program" to do it:

  emails = fetch(emails)
  summaries = map(emails, sandboxed_LLM_summarize)
  output(summaries.concat('\n'))

- The unprivileged LLM then generates the summaries, and the program calling into the unprivileged LLM (which is not an AI) takes those strings and then passes them (sanitized) to `output` (output is also not an AI) and outputs them concatenated together back to the user.

- So, to reiterate, you don't actually get output directly from the privileged LLM. The privileged LLM could write a response with variables that get substituted externally, but you might not even do that. The privileged LLM doesn't directly respond to you, there's a (non-AI) program sitting between you and the privileged LLM that is actually handling output, and that can have untrusted LLM output because it's not an AI and not vulnerable to prompt-injection. So it can do things like just output the concatenated summaries, or it can take the privileged LLMs response and do (deterministic, non-AI) text manipulation/substitution if you really want to.

- And that "output" is now untrusted because it contains "infected" text from the sandboxed LLM, so that output must never be fed back into the system.

I can imagine doing some more complicated stuff if you get clever about variables or have trusted helpers that can give information, but... that's basically the idea behind the limitation here.

Your privileged LLM doesn't ever get to see any output from the unprivileged LLM. All it's really doing is taking human input and translating it on the fly to a list of instructions, and then a non-AI takes the result of whatever the sandboxed LLM's task(s) and sticks it in the output after the privileged LLM is entirely done with everything.

----

Important to note here that this has not gotten rid of prompt injection, all it's done is changed the scope of prompt injection.

I mentioned in my first reply that I think this is kind of fiddly and easy to mess up. As an example, let's say you're coding this up and you decide that for summaries, your sandboxed AI gets all of the messages together in one pass. That would be both cheaper and faster to run and simpler architecture, right? Except it opens you up to a vulnerability, because now an email can change the summary of a different email.

It's easy to imagine someone setting up the API calls so that they're used like so:

  emails = fetch(emails)
  summary = sandboxed_LLM_summarize(emails.concat('\n'))
  output(summary)

So to guard against that, you (should) do a completely separate invocation of the sandboxed LLM for each summary, which still hasn't gotten rid of prompt injection entirely, but it has basically limited it to "an email can only lie about itself", which is not nearly as big of a risk since emails can already do that. But again, limitations, because that's going to end up being a lot slower to run.

taking the example from the article, imagine you have a central personal, household or business LLM that you give general verbal or typed commands to and it intelligently converts those commands to system actions.

you say “give a summary of my most recent three emails”, and the power LLM, instead of unsafely going and doing the summaries itself, accesses/generates a quarantined LLM’s summaries, then displays those summaries to you without actually putting the text through its model

I’m building upon the idea here a little, but let’s say you read the summaries and find them trustworthy, you could then say “reply to email 1 in xyz manner” to the privileged power LLM, which then gives a third LLM with email sending privileges access to summary 1’s file

That's a really good way of putting it. The quarantined outputs are stuck in closed boxes, and the privileged LLM can only ever see the outside of those boxes, not the inside.

> where does the fiddliness come in?

I gave an example in a sibling answer of a common mistake I suspect people would make (having the unprivileged LLM operate on multiple prompts at the same time rather than separately) but it's mostly stuff like that -- I suspect it'll be a little bit tricky with some applications to keep track of what data is "infected" and what data isn't and when it's appropriate to allow that infected data to be mixed together even with itself.

I suspect that for more complicated apps you'll have to be really careful to make sure that there's not some circuitous route where the output of one call gets passed into another one. But it's quite possible I'm overstating the problem. I just worry that someone ends up doing something like extracting a label from the untrusted LLM and sticking into a name or something that the privileged LLM can look at.