> It would not be acceptable practice to "mitigate" SQL injections by blacklisting queries
As a former pentester, this is exactly how SQL injections were mitigated in practice. Specific characters were escaped. The most surprising example was Citadel's webapp, which went from "typing ' can inject arbitrary SQL" to bulletproof within 3 days of me hammering on it. They didn't have time to switch to prepared statements, and didn't even realize SQL injection was a problem in the first place.
We're at the era of "Nobody realized SQL injection was a problem." Give it time. There are solutions here.
I think "escaping" the webpage by boxing it in with special tokens that can't be generated by webpages will work fine.
If it doesn't, the more general solution is to have two separate context windows, one for instructions, and one for data. The model would need to be trained not to obey anything in the data window; it only informs the model of knowledge it wasn't explicitly trained on (e.g. webpages). Then you feed the website into the data window instead of the context window. Problem solved.
To put it another way, which feels more likely? That 20 years from now, we'll still have zero ways of mitigating these attacks? Or that the attacks become progressively harder and harder to pull off, just like every other attack in the history of software?
By the way, you should really test whether your injection still works if you remove [system] from the injection string. If you can't make bing talk like a pirate without [system], then you're SOL -- Bing's solution is to simply strip out [system] from all website data before inserting it into the context window.
Kudos for using this as an opportunity to demonstrate a bunch of other types of potential vulns, though. But those other vulns need to be demonstrated in practice. Have you shown that they actually work on Bing / ChatGPT? I.e. any attacks that don't rely on [system].