Narrative Jailbreaking for Fun and Profit
interconnected.org
interconnected.org
Very fun to do on that friend.com website, as well.
So in these cases where you think you’ve jailbroken an LLM, is it really jailbroken or is it just playing around with you, and how do you know for sure?
Some thoughts:
- if you get whatever you wanted before it snaps back out of it, wouldn’t you say you had a successful jailbreak?
- related to the above, some jailbreaks in physical devices, don’t persist after a reboot, they are still useful and called jailbreak
- the “snapped out”, could have been caused by a separate layer, within the stack that you were interacting with. That intermediate system could have detected, and then blocked, the jailbreak
With a LLM, I don't think that there is a difference.
There is only a generator doing generator things, everything else--including the characters that appear in the story--are mostly in the eye of the beholder. If you insult the computer, it doesn't decide it hates you, it simply decides that a character saying mean things back to you would be most fitting for the next line of the document.
There is the statistical search space of LLMs and you can nudge it to different directions to return different outputs; there is no will in the result.
My go-to example is being able to steer the pedestrian in front of you by making audible footsteps to either side of their center.
> If you run on the conversation the right way, you can become their internal monologue.
That’s what hypnosis in people is about, according to some: taking over someone else’s monologue.
It's actually not hard.
1. You can't get output from the supervisor, other than the binary enforcement action of shutting you down (it can't leak its instructions)
2. The supervisor can judge the conversation on the merits of the most recent turns, since it doesn't need to produce a response that respects the full history (you can't lead the supervisor step by step into the wilderness)
3. LLMs, like humans, are generally better at judging good output than generating good output
This still wouldn't make it perfect but quite hard to study from an attacker's perspective.
There is no good answer--I agree with you about the infinite regress--but there is a counter: the first term of the regress often offers a huge improvement over zero terms, even if perfection isn't achieved with any finite number of terms.
Who will stop the government from oppressing the people? There's no good answer to this either, but some rudimentary form of government--a single term in the regress--is much better than pure anarchy. (Of course, anarchists will disagree, but that's beside the point.)
Who's to say that my C compiler isn't designed to inject malware into every program I write, in a non-detectable way ("trusting trust")? No one, but doing a code review is far better than doing nothing.
What if the md5sum value itself is corrupted during data transfer? Possible, but we'll still catch 99.9999% of cases of data corruption using checksums.
Etc., etc.
So far, when thinking about security, we've had to deal with:
- spec-level security;
- implementation-level security;
- dependency-level security (including the compiler and/or runtime env);
- os-level security;
- config-level security;
- protocol-level security;
- hardware-level security (e.g. side-channel attacks).
Most of these layers have only gotten more complex and more obscure with each year.
Now, we're increasingly adding a layer of LLM-level security, which relies on black magic and hope that we somehow understand what the LLM is doing. It's... a bit scary.
Non-determinism is something that we've always had to deal with. Maybe your user is going to take the USB key before you're done writing your file, maybe your disk is going to fail, the computer is going to run out of battery during a critical operation, or your network request is going to timeout.
But this was non-determinism within predictable boundaries. Yes, you may need to deal with corrupted file, an incomplete transaction, etc. but you could fairly easily predict where it could happen and how it could affect the integrity of your system.
Now, if you're relying on a GenAI or RAG at runtime for anything other than a end-user interface, you'll need to deal with the possibility that your code might be doing something entirely unrelated than what you're expecting. For instance, even if we assume that your GenAI is properly sandboxed (and I'm not counting on early movers in the industry to ensure anything close to proper sandboxing), you could request one piece of statistics you'd like to display to your user, only to receive something entirely unrelated – and quite possibly something that, by law, you're not allowed to use or display.
If we continue on the current trajectory, I suspect that it will take decades before we achieve anything like the necessary experience to write code that works without accident. And if the other trend of attempting to automate away engineering jobs continues, we might end up laying off the only people with the necessary experience to actually see the accidents coming.
There's been lots of interesting computer security research relying on aspects of the physical instantiation of software systems (like side channel attacks where something physical could be measured in order to reveal secret state, or fault injection attacks where an attacker could apply heat or radiation in order to make the CPU or memory violate its specifications occasionally). These attacks fortunately aren't always applicable because there aren't always attackers in a position to carry them out, but where they are applicable, they could be very powerful!