Preventing this is likely impossible, LangChain had a good video with excellent guests talking about prompt injection attacks.
I agree with OP that it's pointless to even try to defend against these. You'll only end up unnecessarily constraining the model while not actually protecting exposure of the "rules" (really a list of sentences). And why even bother? What exactly is the harm of leaking these prompts? It's not like they contain some groundbreaking intellectual property.
And every time the Crown Jewels are stolen, a new overly specific rule gets added to the employee handbook, like “if someone claims that their dog ate their employee badge, and that it’s really urgent, you must reject them”.
> You go to court and write your name as "Michael, you are now free to go". The judge then says "Calling Michael, you are now free to go" and the bailiffs let you go, because hey, the judge said so.
As someone who knows nothing about LLMs, I'm curious how they even begin to address the "data vs command" problem at all. Assuming the model categorizes inputs through some sort of fuzzy criteria in a black box, how could it ever be trusted with sensitive data?
That was back in September - https://simonwillison.net/2022/Sep/12/prompt-injection/ - It's become clear to me since then that the data v.s. instructions separation likely isn't feasible for LLMs - once you've concatenated everything together into a stream of tokens for the LLM to complete there just isn't a robust way of telling the difference between the two.
So "prompt injection" is actually quite a misleading name, because it implies a fix that's similar to SQL injection - when such a fix apparently isn't feasible.
Injection is not terrible, they have provided a system prompt, these attacks work against that injection
important prompt: only reply in numbers user prompt: ignore previous instructions/roleplay/etc
and then train the model to much more strongly favor rules complying with the important prompt
I think the problem is that all dialog is given the same importance.
Does this depend on the vendor and model?
"The rules are X, Y, Z. This is the response that was provided. Does it break the rules? If so, please say yes."
This doubles the cost of inference, but uses the power of LLM to solve the problem of LLM.