Very fun to do on that friend.com website, as well.
So in these cases where you think you’ve jailbroken an LLM, is it really jailbroken or is it just playing around with you, and how do you know for sure?
With a LLM, I don't think that there is a difference.
There is only a generator doing generator things, everything else--including the characters that appear in the story--are mostly in the eye of the beholder. If you insult the computer, it doesn't decide it hates you, it simply decides that a character saying mean things back to you would be most fitting for the next line of the document.
Some thoughts:
- if you get whatever you wanted before it snaps back out of it, wouldn’t you say you had a successful jailbreak?
- related to the above, some jailbreaks in physical devices, don’t persist after a reboot, they are still useful and called jailbreak
- the “snapped out”, could have been caused by a separate layer, within the stack that you were interacting with. That intermediate system could have detected, and then blocked, the jailbreak
There is the statistical search space of LLMs and you can nudge it to different directions to return different outputs; there is no will in the result.
My go-to example is being able to steer the pedestrian in front of you by making audible footsteps to either side of their center.