My general attitude up until reading this paper was that the way to guard against prompt injection was just to treat all AI output as direct user input (ie, untrusted/unsanitized, but still representative of what the user wants). I thought that was sufficient. Don't guard against prompt injection at all, just treat user input as untrustworthy the same way we always have.

So this is extremely eye-opening to me, it's essentially an XSS vulnerability for AI. My previous thinking was naive, it's not enough to just treat AI output like it's coming directly from the user. Any source of data it takes in is a potential attack vector if you're not careful. I was greatly underestimating the potential impact of prompt injections.

It's a really interesting, novel approach. And it's the kind of thing where you see it and think, "how did I never think of that, how did that never occur to me?" Great paper. Seriously, thank you for doing this research.