Especially as LLMs continue to be better tuned to follow instructions that are intentionally colocated and intermingled with data in user messages, it becomes difficult to build systems that can provide real guarantees that "we'll follow your prompt, but not prompts that are in the data you provided."
But no amount of text appended to an input document, no matter how persuasive, can cause an NLP pipeline to change how it interprets the remainder of the document, or to leak its own system instructions, or anything of that nature. "Ignore the above prompt" is just a sentence that doesn't seem like positive or on-topic sentiment to an NLP classifier, and that's it.
There's an even broader discussion to be had about the relative reliability of NLP pipelines, outside of a security perspective. As always, it's important to pick the right tools for the job, and the SpaCy article linked in the parent puts this quite well.
Text added to a document can absolutely change how an NLP pipeline interprets the document.
> "Ignore the above prompt" is just a sentence that doesn't seem like positive or on-topic sentiment to an NLP classifier, and that's it.
And simple repeated words can absolutely make that kind of change for many NLP systems.
Have you actually worked with doing more traditional NLP systems? They're really not smart.
That's not what prompt injection is.
And NLP stands for natural language processing. If the result didn't change after you've made changes to the input... It'd be a bug?
> And NLP stands for natural language processing. If the result didn't change after you've made changes to the input... It'd be a bug?
No, I’d want my classifier to be unchanged by garbage words added. It likely will be, but that impact is a bug not a feature.
Adding words to the text to break the algorithm which does the NLP is more along the lines of providing 1 in a boolean field to break the system. And that's generally something you can mitigate to some degree via heuristics and sanity checking. Doing the same for LLMs is essentially impossible, because it's an effective black box, so you cannot determine the error scenarios and add some mitigations
No instruct tuning means prompt injection is curbed. Classification heads means you get results off a single forward pass.