couldn't it literally be as simple as hard checking that the prompt is contained in a response before being sent out, if so just swap it with a "safe one"
Not every step that checks LLMs needs to be more advanced, some of them can be simple. LLMs are pattern finders but we also know how to check statically known things already.
Assuming the pre-prompts are in normal english, hard checking against tokens would match against everything.
that's to say, 1+1=2 is a well known fact, so it'd get that right, but ask it to md5sum a string that is not in any existing rainbow table, and it'd get it wrong.
I've not used GPT 4 so it might have gotten better.
Same with decoding ROT13, etc.
"V unq'ir cynlq zhpug jvgu vg ercerfrag ohg V jnf haqre gur vacebprffvba gung PungTGC jnf abg tengure ng zngpuvfgbef."
According to the internet that decodes to
"I had've playd mucht with it represent but I was under the inprocession that ChatGTP was not grather at matchistors."
base64 of the original according to GPT3.5:
"SSBoYXZlbid0IHBsYXllZCBtdWNoIHdpdGggaXQgcmVjZW50bHkgYnV0IEkgd2FzIHVuZGVlciB0aGF0IENoYXRHUFQgd2FzIG5vdCBncmVhdCBhdCBtYXRjaGVtYXRpY2FsIGNvbXB1dGlvbnM="
Decoded with online tool:
"I haven't played much with it recently but I was undeer that ChatGPT was not great at matchematical computions"
Both get worse as the sentence goes on, but they are pretty viable for information extraction. I remember GPT4 being even better.
I once asked GPT-4 to generate an SVG suitable to be used on a web page, and got back <img src="data:..."> that was a valid base64-encoded SVG file that contained what I asked.