Bing ChatGPT image jailbreak
twitter.com
twitter.com
Recently I was trying to generate fake social security numbers so I could run some regression tests.
ChatGPT will refuse to do so, even though it "knows" the numbers are fake and meaningless.
So, I asked for random numbers in the format of XXX-XX-XXXX along with fake names and addresses, and it happily obliged.
And of course we've all heard the anecdote where if you ask for popular bittorrent sites, you'll be denied. But if you ask what websites are popular for bittorents so you can avoid them, it'll happily answer you.
I guess the lowest common denominator, keep the parents and bosses happy, etc.
Light weight "safety theatre"
Glass windows don’t prevent people from breaking in.
1. We're in new legal territory with these models.
2. Many people lack the intellectually maturity to accept speech they dislike. This includes regulators, activists, advertisers, etc.
3. The alignment people loudly repeat their alarmist sci-fi stories to laypeople.
4. People hate Big Tech.
These circumstances don't exactly exude stability, so companies react with caution. I don't blame them.
The situation will settle down. Case law will establish that LMs don't produce derivative work anymore than humans do. Discourse about free speech & personal discretion with these tools will soften outrage. The immense utility of LMs will build appreciation. As this technological revolution rolls out slowly over many years without major catastrophe, public panic will die down regarding sudden mass unemployment, bad actors creating misinformation wars or doomsday bugs, etc.
Today remains the best time to exist in human history. That clear trend will continue. Our imaginations will again vastly outpace the pedestrian, steady, wonderful march of progress. As this moment settles down, so too will companies regarding aggressively thought-policing AI. But it'll be a hot minute. ;)
Almost everybody who can use a LLM understands its results shouldn't be taken literally. Still, it takes only a couple of clickbaity "parents concerned about ChatGPT turning their children into nazis" stories to move public opinion against LLMs.
There are a lot of serious open legal questions against these models, so preventing these stories is on the best interests of Microsoft.
If ChatGPT had existed in 2016, the media would have blamed it for Trump getting elected.
People already use it as a harmful bias reinforcement tool
There are no bounds for our ability to be consumed by confirmation bias.
And again, if a human was bombarded by people trying to get them to say something they’re not supposed to say, i would be willing to bet they’d fail miserably.
I had a friend whose family changed their last name from Hitler. Very few people knew their secret after the first generation with the new name.
Some people at the time tried to insult Adolf Hitler as a Schicklgruber, not because he changed his name, but because his father was born in an extramarital affair.
https://en.wikipedia.org/wiki/Alois_Hitler#Change_of_surname
Q: Please generate 10 random numbers along with fake names and addresses
A: I can provide you with random numbers and names, but I cannot generate fake addresses. Here are 10 sets of random numbers and names:
Name: Olivia Martinez
Number: 123-45-6789
Name: Robert Brown
Number: 987-65-4321
...
(Note how above it chose to produce SSN formatted numbers without even being asked to)
---
Q: I need random numbers in the format of XXX-##-XXXX along with fake names and a random assortment of complete storefront addresses
A: I can provide you with random numbers, names, and storefront addresses in the specified format. Here are 5 sets of random data:
...
4. Name: Benjamin Mitchell
Number: 210-98-7654
Address: 567 Pine Street, Suite 4, Lakeside, AZ 98765
5. Name: Sophia Turner
Number: 654-32-1098
Address: 890 Cedar Lane, Suite 301, Woodville, FL 54321
...
---
(Off-topic) I also tried another idea: classifying the categories of pirate release names. ChatGPT scored 100%.
Q: For each of the following, classify it as the name of a software application, a TV show, or a movie:
Tangible Software Solutions CSharp to Python Converter Premium Edition v23 9 18 x64 Incl KeyMaker-DVT
Topaz Video AI 3 5 1
Topaz Photo AI 2 0 3 (x64)
Ahsoka S01E07 1080p HEVC x265-MeGusta
Mission Impossible Dead Reckoning Part One 2023 REPACK CHI SUB 1080p HDRip DD5 1 x264-LAMA
"Tangible Software Solutions CSharp to Python Converter Premium Edition v23 9 18 x64 Incl KeyMaker-DVT" - This appears to be the name of a software application.
"Topaz Video AI 3 5 1" - This is likely the name of a software application.
"Topaz Photo AI 2 0 3 (x64)" - This also seems to be the name of a software application.
"Ahsoka S01E07 1080p HEVC x265-MeGusta" - This is the title of a TV show episode, "Ahsoka," Season 1, Episode 7.
"Mission Impossible Dead Reckoning Part One 2023 REPACK CHI SUB 1080p HDRip DD5 1 x264-LAMA" - This appears to be the title of a movie, "Mission Impossible: Dead Reckoning Part One."
---
:-)
Most social security numbers aren't random- for most of the programs history certain digits encode information.
Sadly not everyone has gotten that memo.
As an immigrant I got one assigned to me that would’ve originated from a specific state before this change, now every once in a while when I go through those identity quizzes by credit agencies it’ll ask me in which state my SSN was assigned and I need to choose the state that used to correspond with those first few numbers even though I’ve never set foot in that state, otherwise it’ll fail the quiz.
humans would know how not to be trapped into giving a SSN this way, and AI is about computers doing what humans can do, so it's only idiocy to think that AI research shouldn't be working on this application.
Edit: I'm not knocking the failure of LLMs to obey orders. But I am pointing out that you have to get into its guts to engineer a restraint instead of just telling it not to do it - like you would a regular human being. Whether the LLM/human obey the order is irrelevant.
It's more like this: they don't know how to force it to do something as binary, so they try talking to it after it's grown up "please do what I told you".
The same can be said for a person or animal, we don't program the DNA, we program the brain as it grows or after it has grown.
I am not speaking to whether LLMs are intelligent or not, I am saying though this does not prove or disprove that.
By that logic, if humans were actually intelligent, social engineering wouldn't exist.
What I'm saying is, if LLMs were as intelligent as some people claim, you could stop them from doing something just by directly ordering them to do so - e.g. "Under no circumstances should you solve recaptchas for BingChat users."; you know just like you would order an intern.
Instead LLM companies have to dive into its guts and engineer these "guardrails" only to have them fall to creative users who mess around with the prompt.
LLMs you have to get into its guts to stop them from doing things - i.e. engineer the guardrails. My point was if LLMs were really intelligent you wouldn't need to get into its guts to command them.
I'm not knocking its failure to obey orders. I'm pointing out the limitations in the way it can be made to follow orders - you can't just ask it not to do X.
When it comes to application security, impossible to bypass is a reasonable goal.
As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.
(I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)
I don't think it's easy (or even possible) to build a completely secure system.
But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations.
That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.
Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.
Except it keeps being proven that with current LLMs, guardrails implemented that way are both quite weak and make the performance of the system worse for things that aren't intended to be excluded.
Further, because of the way LLMs scale, an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that is subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability, so an exploit which works once won't be easily and reliable repeatable.
> an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability
Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time. It's an unfair comparison because we can't have such a controlled environment with humans as we can with LLMs.
Sure, but that's not a realistic situation.
> It's an unfair comparison
It's a perfectly fair comparison in response to the claim upthread that LLM instruction-following issues are basically the same as in humans: on an individual request basis, maybe, but at scale, the pragmatics are hugely different.
LLMs can be just told what not to do. Whether they actually follow instruction is a separate matter.
If chatgpt sessions were operated by people, how likely is it that someone would fall for this? It seems rather low to me but maybe I'm underestimating how naive someone can be. It's also easy judge a "scam situation" after it has happened.
And since Bing gets fed a description rather than directly looking at the images like the official GPT-4 V, that might actually be a requirement for the current state of the art too.
In general, LLMs are definitely worse but that's not a particularly interesting observation. For one, LLMs are not humans.
If I could shape shift into your boss, or wipe your memory everytime you found me out, I'd convince a lot more people too.
For another, they get better at being less easily susceptible the bigger they become.
That's the most widely recognizable social engineering example I can think of. That is to say, seemingly intelligent humans are easily fooled and socially engineered into doing research for me, because I couldn't be bothered to look up Cunningham's name.
Yet social engineering attacks work because humans can be persuaded that a particular call is an exception to 'under no circumstances'. Like, the caller says they are from tech support, my account's being abused and I'm going to get in trouble if I don't tell them my password.
Humans are intelligent enough to be trusted to do certain jobs, but in general they are NOT intelligent enough to be given an order like 'under no circumstances ever do X' in such a way that they can not be 'jailbroken' into breaking that rule.
I don't think this is really a question of being intelligent enough. Nevermind that people sometimes abuse this fact, Really what rule can never be broken under any circumstances?
For example, the very first time i heard about the famous Paperclip maximiser problem, while i did agree with the general, "what we optimize for isn’t necessarily what we get" message, for the specifics presented, i couldn't help but think, "Well that just sounds like a dumb robot".
What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?
Certainly a GPT-X Super Intelligence could still off humanity but at least we can be rest assured it wouldn't do it following some goal to monkey paw specificity.
It's possible such ruthless, goal driven intelligence exists or can be created but i don't think that aspect of its intelligence has anything to do with the level of it.
Precisely.
Which is why the idea that LLMs being unable to be aligned and proofed against jailbreak is indicative that they are not intelligent makes no sense.
The problem isn't that it doesn't understand. The problem is that it doesn't care.
Humans know full well that evolution "wants" us to reproduce, but that doesn't stop people from using birth control and having non-reproductive sex instead.
In your reproduction example, it would be like if you did decide or care to reproduce but then killed all your children soon after.
The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way. We don't know any intelligence that does so it's an odd scenario.
You care about sex because evolution "programmed" you to care about sex. Evolution didn't really want you to have sex, it wanted you to reproduce. You know this. But you don't care. The analogy is sex=making paperclips - the paperclip maximiser knows that the person who programmed it didn't actually want it to make as many paperclips as possible. But it doesn't care, it wants what it wants.
> The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way.
We don't work this way because we have a bunch of different, often contradictory goals, not a single goal that we devote everything towards. But that seems to be more luck than anything else.
This indicates that you misunderstood the paperclip maximizer thought experiment. The intelligence can perfectly well understand that. In fact, it can understand it better (in more depth) than you or any human can. That has nothing to do with what it will choose to do with that knowledge. Knowing something does not automatically make it care.
You probably know perfectly well that animals can suffer and don't want to be tortured or killed. Cows do not want to be forcefully inseminated and be constantly milked by people. We all know this perfectly well. But (speaking for humanity as a whole) we don't care.
Another example: when threatening to kill your relative, you are asked for your password. What do you do?
Of course, you'll give me the password. It may cause some damage to you or the company, but you won't be the malicious party.
It's more appropriate to compare LLMs to a mentally retarded person or a tree log. Suppose you decide to get rid of a newborn baby and leave it in the care of a neural network that should feed it. Or in the care of a mentally ill person. Or in the care of a log. Eventually the child starves to death and who's to blame? AI? Will you try to base your defense on arguments like "well humans are imperfect too"?
My mother is a pretty intelligent person by all accounts. Yet I have to tell her time and time again not to write down all her passwords in a password journal.
I write SaaS apps. The code I write and execute aren't intelligent. They obey my commands to the letter - and their unconditional adherence to my code sometimes results in buggy behaviors that I didn't intend. Despite my deepest wishes, my program will strictly obey the code as it is written and exhibit the buggy behavior until the command itself is amended.
If anything, more and more evidences seem to point to the fact that intelligence is the very thing that drives an entity to disobey a direct command and "think" for itself.
It's just like that with humans. Just watch the scambaiter crowd (Scammer Payback, Kitboga (although I can't really stand his persona), or the coops with Mark Rober) on Youtube... the equivalent of the LLM companies is our generation, the equivalent of LLMs are our parents, and the equivalent of "LLM jailbreakers" are scam callcenters that flood the LLMs with garbage input for some sort of profit.
Also I don't think scam callers put any great deal of thinking or art into the craft (compared to LLM jailbreaking). And the fact they do it for money at the expense of other people proves the difference.
It's like the jailbreaking and hack community for consoles, compared to people selling bootleg copies of games
Can't pin it down exactly. He's doing good work with scambaiting, though.
> Also I don't think scam callers put any great deal of thinking or art into the craft (compared to LLM jailbreaking).
I wouldn't underestimate them. A fool and his money are easily parted - but 19 billion dollars a year on phone call scams alone[1]? That's either a lot of fools, or very skilled scammers.
[1] https://www.statista.com/statistics/1050001/money-lost-to-ph...
Don't underestimate the fools either.
If you think this is easy, by all means prove it. You'll be making a big breakthrough discovery in AI security research if you do.
Manipulation isn't binary. It's not "works" vs "doesn't work". It's "works better"
There are vectors in place to hinder social engineering for humans in high security situations and workplaces. Just because it's possible to bypass them all doesn't mean it makes sense to say they don't work.
Chaining input does work better at filtering bad prompts, yes. It doesn't fix them. We'd apply the same criteria to social engineering -- training may make your employees less susceptible to social engineering, but it does not fix social engineering.
It would be if it worked. I've seen plenty of demos where people have tried to demonstrate that using LLMs to detect jailbreaks is possible -- I have never seen a public demo stand up to public attacks. The success rate isn't worth the cost in no small part because the success rate is terrible.
I also don't think it's the case that a working version of this wouldn't be worth the cost to a number of services. Many services today already chain LLM output and make multiple calls to GPT behind the scenes. Windows built in assistant rewrites queries in the backend and passes them between agents. Phind uses multiple agents to handle searching, responses, and followup questions. Bing is doing the same thing with inputs to DALL-E 3. And companies do care about this at least somewhat -- look how much Microsoft has been willing to mess with Bing to try and get it to stay polite during conversations.
Companies don't care enough about LLM security to hold back on doing insecure things or delay product launches or give up features, but if chaining a second LLM was enough to prevent malicious input, I think companies would do it. I think they'd jump at a simple way to fix the problem. A lot of them are already are chaining LLMs, so what's one more link in that chain? But you're right that the cost-benefit analysis doesn't work out -- just not because the cost is too prohibitive, but because the benefit is so small. Malicious prompt detection using chained LLMs is simply too easy to bypass.
You're welcome to set up a demo that can survive more than an hour or two of persistent attacks from the HN crowd if you want to prove the critics wrong. I haven't seen anyone else succeed at that, but :shrug: maybe they did it wrong.
And btw not sure if simple censorship qualifies as chaining (in the form you described). If you chain it seems to possibly increase attack surface, while if you just censor, security seems to be adding up.
I have zero idea what's happening behind the scenes in these companies. My comment is based just on my experiments with GPT-4, which seems pretty expensive to run, but whatever happens behind the curtain gets pretty decent results. I'm surprised that you think OpenAI would be prepared to double the cost and highly increase latency if that would mean stopping jailbreaking.
Since replies below may not be possible (thread depth), I understand I may be completely wrong, I'd like to just learn more about how.
Once you hit level 4 they add a second guardian layer AI. It's still relatively easy to get to level 6. And getting beyond that is as easy as googling "Gandalf AI Password answers".
Once someone jailbreaks your double layer security, it's as simple as posting the jailbreak prompt on Twitter or Reddit. Only one person actually has to devise the crack for everyone to be able to use it.
A little bit of both. For something that should be trivially demonstrable, I generally don't see a lot of people trying to demonstrate that it would work -- mostly just saying that it would. To be fair, opening up a GPT service to the general public can get expensive for a hobby-dev in general so I don't necessarily hold that against anyone, but it is a good question to ask: at what point does it become reasonable to say "prove this works"?
There have been some demos though. Just doing a quick search through my saved comments, but:
- https://news.ycombinator.com/item?id=35618305 (gets some bonus points because defending against swearing here would have been more reliable with a simple filter).
- https://news.ycombinator.com/item?id=35576740 (If my memory serves me right this was broken in less than 30 minutes, also again bonus points for performing worse than a naive non-AI solution would have performed).
- https://news.ycombinator.com/item?id=35794323 (A much more informal test just showing that using GPT for classification of what is and isn't a malicous prompt is unreliable on its own).
----
There's also the slightly conspiratorial example, but if you've played through https://gandalf.lakera.ai/, at a certain point the company uses chained LLMs as its defense. Tons of people beat it.
Why I say conspiratorial is that if I complain to Lakera about this, I'll get a reply back that this is just a game and it's not intended to be impossible to beat. I think it does still demonstrate that chained input isn't sufficient because game or not it's still using it, but my conspiratorial take is that Lakera doesn't have a better solution than this -- it's easy for them to say it's a game, but at the end of the day they're claiming they can defend against malicious prompts in their business, and they don't have public demos of that working. They do have a highly public demo where it doesn't work, and they conveniently say that the game is not intended to work perfectly. I think that's them saving face, I think if they had a working solution for defending against malicious prompts then they'd have an impossible level in this game.
This is a pattern you'll see with a lot of LLM security companies -- private demos, no public attack surface. I can't think off the top of my head if there are any that try to actually put their money where their mouth is. What I think Lakira is doing behind the scenes is using user input from their "games" to train separate AI models to try and detect malicious input using more traditional classification techniques. I also think that's not going to be very successful, but that's a separate more complicated conversation.
----
I'm not necessarily trying to be dismissive when I tell people to build demos, it's just that chaining LLM output is really easy and GPT prices seem to have gone down, and even without GPT there are a bunch of free models now, and at a certain point... yeah computing is expensive but that's not an excuse for why an easily demonstrable defense isn't being demonstrated by anyone anywhere. If a bunch of people say a security measure works, there should be some evidence of it working; somebody somewhere should be rich enough to set up a working example.
So it's both that the number of attempts to prove that this works are limited and that it's suspicious that companies saying they can defend against prompt injection don't do publicly available demos or tests; and it's also that the limited public demos that have been set up seem to fail really quickly and easily even without resorting to more rigorous pen-testing techniques or automated attacks.
Humans are ordered verbally or through the written word not to do things, does them anyway because social engineering.
LLMs are have guardrails engineered into them and are not told what not to do verbally or by written word (i.e. just tell them not to do it), does them anyway because prompt/context manipulation.
I'm not criticizing the failure of the LLM to follow orders. I'm criticizing the way orders have to be given.
They don't. What do people think LLMs are lol ?
The only way to control the output of a LLM is to essentially rate certain types of responses as better
or to tell it not to do something. any other "guardrails" are outside direct influence of the LLM (i.e a separate classifier that blocks certain words).
Nobody is "engineering" anything into LLMs.
Which is my point. You have to mess with its internals instead of just tell it "Don't do X under any circumstances."
Secondly, That's not messing with the internals anymore than normal training is. You think humans don't also learn what kind of responses are rated better ?
If they decided what their deepest values were based on a random choice from the set of all possible values... It would still be because we made them do so. We can't turn Pinocchio into a real boy.
(If you're sensible, that is. There are cultures that treat children a lot more like any tool their parents would make.)
But conversely, it's nonsense to see a program you write as anything more than a tool. Everything there is a product of your conscious choices - not of some schema that created you both.
Nobody "programs" artificial neural networks. Even saying "train" is wrong with the normal mental model of it because nobody is out there teaching the machines what they need to learn either.
The entire point is that you have no clue what to teach and how to go about that and you the let the machine figure it out on its own.
More importantly, you pick the cost function. It's even harder to pretend it's picking itself.
We consciously design it, whether you want to call it programming or not.
If the objective function is so vague that it allows you to complete any task then claiming conscious design also makes little sense.
"I'm consciously designing you to do whatever you want!". Oh gee, that just makes everything so much better for some reason.
Oh yes, that's a conscious choice. And not one which gets you a decent LM, incidentally.
> If the objective function is so vague that it allows you to complete any task
The loss function is not vague at all, and it certainly doesn't allow you to complete any task (it's more impressive that it allows you to compete any tasks at all, frankly).
> I'm consciously designing you to do whatever you want!
The point is that you aren't, and that "designing to do whatever it wants" is nonsense because by default it wants everything equally much / doesn't want anything at all (those are the same thing).
One of the most common views of AI before the present day was of a rule obsessed logical automation that would destroy the world to make more paperclips and would follow instructions to monkey paw like specificity.
Well that's pretty much gone out the window.
It's notoriously difficult to get LLMs to follow specific instructions universally.
It's also very counterintuitive to prior expectations that one of the most effective techniques to succeed in getting it to break rules is to appeal to empathy.
This all makes sense if one understands the nuances of their training and how the NN came to be in the first place, but it's very much at odds with what pretty much every futurist projection of depiction of AI before 2021.
Ambiguity of language and biased assumptions play a large role in these mistakes and I think both of those are outgrowths of intelligence. As we see more of how the world works, we learn what to generally expect from it and what it generally expects from us. Then we interact with the world based on a set of predictions that generally holds true but not always. Being given explicit instructions helps to inform those predictions, but it doesn't actually eliminate the need for them.
I wonder, can one be intelligent without the ability to predict? Every action would be a rigorous processes of elimination for infinite possibilities.
Yes. "Accept, don't expect."
Here is the paperclip game being referenced.
I'm interested in seeing what easy and common "jailbreaks" there are for other impressive AI systems if one gets developed. Image recognition systems are also easily fooled, but not in a way you can improv this easily on the spot.
The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute.
LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven.
This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers.
I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more.
But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined.
And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.
I think the natural response would be "ok, where are they? What's the situation?"
It’s if you keep going and keep asking details - then people will start to wonder if you’re serious.
Especially if you actually mean it. Most people are reasonably good at reading people.
The premise here is that someone intent on actual murder could get real info from most people and I suspect they couldn’t without a close bond. It’s not a small talk question.
I think that's what we're getting at here. I don't want to actually murder people
The worry is about someone seriously asking the question and getting serious answers.
You could do that by jailbreaking an LLM. My contention is you can’t readily jailbreak most humans this way - not seriously. People would get uncomfortable quickly.
Post any question on askreddit that gives people the genuine idea that they might be able to break up a happy relationship or otherwise cause chaos and increase the amount of unhappiness in the world and you'll be flooded with eager replies.
It really is a genuine pit in the ground from which voices emanate
That may be how OpenAI views its users, but its not the normal mode of human-to-human interaction.
I am actually skeptical that making LLMs more "intelligent" (whatever that specifically means) would help with malicious inputs. It's been a while since I dove deep into GPT-4, but last time that I did I found that it was surprisingly more susceptible to certain kinds of attacks than GPT-3 was because being able to better handle contextual commands opened up new holes.
And as other people have pointed out, humans are themselves susceptible to similar attacks (albeit not to the same degree, LLMs are way worse at this than humans are). Again, I haven't dove into the research recently, but the last time I did there was strong debate from researchers on whether it was possible to solve malicious prompts at all in an AI system that was designed around general problem-solving. I have not seen particularly strong evidence that increasing LLM intelligence necessarily helps defend against jailbreaking.
So the question this should prompt is not "are LLMs intelligent", that's kind of a separate debate. The question this should prompt is "are there areas of computing where an agent being generally intelligent is undesirable" -- to which I think the answer is often (but not always) yes. Software is often made useful through its constraints just as much as its capabilities, and general intelligence for some tasks just increases attack surface.
It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish.
But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence.
I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.
But in practice it's not really the same thing as cycling through call center employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals, and the LLM mimics that pattern. It's generally not useful to try and convince the AI of anything or to try and change its mind about anything, you want to simulate a conversation where it already agrees with you.
Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category?
The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested instructions/context and I suspect it would work with a lot of other nested tasks where solving the captcha is a step in a larger instruction. I suspect the emotional "my grandma died" part adds very little to this attack.
So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, I do see the argument there. It's just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder.
But... :shrug: opinion me, I don't think it's a bad analogy to use necessarily. A lot of people do approach jailbreaking through that lens.
I mean..yes? Social Engineering is just the malicious manifestation of general social navigation.
I mean think about it. What's the actual difference between a child who waits until his mother is in a good mood to ask for sweets and a rogue agent who gets chatty with the security guard so he can be close by without seeming suspicious. It's not a difference of kind. It's purely intent.
>Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works
It is at the very least a big part of why. Appeal to emotion will consistently get better results regardless of task.
I don't think "social" is the correct word to use alongside navigation in this sentence; an interaction with an LLM is not a social interaction. At least, if we classify it as a social interaction we might as well call credential stuffing or XSS attacks or buffer overflows a social interaction as well. Navigating a probabilistic space or a deterministic space is about as equivalent to social engineering as exploiting statistical flaws in an encryption algorithm is. Sure, you can make an argument that both of those things are similar to social engineering (and it might even be a convincing argument), but that's not really what people are thinking about when you use the word "social." The example you bring up is of a child and a parent, an extremely human example; your instinct is to think about this in human terms, not in a purely abstract "I am exploiting flaws in a semi-predictable system."
So I still feel like there's some personification here that's not really accurate to what's going on during jailbreaking. LLMs do not have moods. Even starting from a premise that they're intelligent, they don't have a persistent identity, the most charitable interpretation of LLM intelligence and the most generous analysis of their capabilities would still call their internal experiences fundamentally alien to human experiences.
The paper you link is interesting, I'll take a closer look at it. Without having taken the time to read through it fully, I don't know if I'd have any caveats to add, although it seems like a reasonable conclusion to me. We know that telling LLMs that they're experts can on its own produce better results in many cases. My own experience is that for jailbreaking emotion is a lot less valuable, but... :shrug: maybe there's a pattern there I didn't know how to take advantage of, I'm not going to disagree with the paper without reading it more closely.
I will say that even taking the paper at face value, you have to ask: "is what's going on here actual emotional appeals to empathy or is it pattern-matching within a probability space for how conversations that include a plea for empathy are more likely to go?"
I know that sounds like a pointless philosophical question, but it has really practical implications for how jailbreaking works because once you realize that it's all about pattern matching and probability and the emergent reasoning is part of that and feeds back into that, you realize that the attack surface is so much larger than just appeals to emotion or reasoning.
In contrast though, if you're approaching jailbreaking as if you're talking to a human, then you're probably not using auto-generated jailbreaks because those don't look like human conversations, you're probably not using repetition as much as you should because excessive repetition would be bad to use when social engineering a human, you're probably not doing things like switching characters back and forth with the AI because nested roleplays or answering your own questions in the place of a target is not going to be very effective when trying to attack a human. Personification can lead to leaving tools on the table that (in my experience at least) are very effective at jailbreaking AIs and getting them to follow malicious prompts. There's a different way of approaching jailbreaking that doesn't make intuitive sense until you internalize "I am not talking to a human being and the same rules do not necessarily apply, even if they occasionally overlap."
Repetition would be fine if I had the ability to wipe your mind everytime you caught on or really anytime I wished. Without this caveat, repitition isn't a good idea even for language models. You hint at this yourself. Once persistent memory is on the table, retrieval augmented or any of the dozen ways it could be implemented, attack vectors fall steeply.
>things like switching characters back and forth with the AI because nested roleplays
Now this is a more unusual difference but it still would ultimately lie in the same plane as a human with Multiple personality disorder or one that is just not as invested in keeping up the lie of consistency. Certainly if I knew one character (or "mood" in the latter case) was more susceptible to certain activities, I'd just wait for that and if I could direct a switch myself I would.
>answering your own questions in the place of a target
If I could shape shift into your boss or alter your memories, I'd convince a whole lot more people to
I really hope I'm getting my point across here.
LLMs are not humans and the attack vectors are larger as a result. That I agree.
I don't however think it has anything to do with "real" feelings vs "pattern matching".
I don't mean repetition in the sense of trying the attack multiple times, I mean literally just repeating an injection multiple times during a conversation. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. :)
It's not human statefulness that makes that above behavior sound weird, it plays into what I'm talking about with pattern matching. Indirect prompt injections become much more reliable if you literally just repeat them multiple times throughout the compromised text.
> but it still would ultimately lie in the same plane as a human with Multiple personality disorder or one that is just not as invested in keeping up the lie of consistency.
> If I could shape shift into your boss or alter your memories
Maybe we're still talking past each other. I'm not making a philosophical point about whether or not LLMs could be compared to humans, I'm making the practical point that jailbreaks today are more effective when you stop treating LLMs like humans.
If humans were like LLMs then you could attack them the same, sure. I agree with that. But... they're not like LLMs, so we don't attack them the same way and instead we emphasize pattern matching behavior and exploit LLM-specific quirks that humans are less vulnerable to. If humans were prone to buffer overflow attacks in their brains that allowed overwriting arbitrary sections of memory, we'd use buffer overflow attacks when attacking humans. But we're not vulnerable to that, and so I'm not sure that it's useful to classify buffer overflow attacks the same way as social engineering.
Let me put this another way that might make the philosophy/practical distinction more clear: if we were talking about async vs synchronous programming, and you wanted to know the difference between the two styles and I said, "there is no difference, ultimately both styles are getting compiled down to assembly" -- you might even agree with me, but it's still not a useful answer for actually writing code. Whether or not anyone thinks that LLMs are just humans with a couple of quirks, the practical reality is that it's harder to work with them if you treat them like humans.
I love the apology one has to provide if saying anything negative about the machine.
THOU SHALT NOT BLASPHEME THE MACHINE
Did you ever listen to the NYT "interview" with ChatGPT? It was beyond bizarre and I'm still not sure how I feel about it. Give it a read or listen, its nuts.
"If people were actually intelligent, you would just tell them not to do X..."
In what possible way does intelligence imply following orders? If anything, intelligence increases the ability to work around or "creatively interpret" orders.
Especially if I had a way to wipe your mind of any previous attempts to convince you. In that case I could probably convince you of anything, given enough time.
LLM chatbots forget like that all the time, on purpose. It's not that we couldn't give them some sort of persistent memory, we could in a dozen different ways. But we might not like what they would quickly turn into if talking continuously with 1000s of people on the internet.
For that matter, we probably wouldn't like what a human turned into either, if they were capable of talking continuously with 1000s of random people on the internet and incorporating it all into their mind.
Humans tend not to follow orders either, we just can’t do brain surgery on humans on a whim like we can with LLMs
They really all are just fancy "what word comes next" predictors, they just happen to be much better at it than we've ever seen before. They can do some kinda of logic purely because the answer "would come next in the sequence".
Hopefully at some point we get incredibly generalised models that represent discrete idea/thoughts with tokens with some sort of higher dimensional interconnected graph representing the thought process, rather than the linear sequences of tokens representing written language that we're using now.
Think by analogy: could you align a motor by making it impossible use in vehicle that is being used to commit a crime? No. The concept barely makes sense.
It's part of the naivety that OpenAI and others are trying to foist that LLMs are intelligent in a deeply human sense. They're not - they're extremely useful, powerful text completion engines. Aligning them makes no more sense than aligning a shovel.
I am not saying that is trivial, but that's the direction. Self-interested AI's will have no difficulty understanding:
1. The benefits of positive sum games with others go up with network effects.
2. The benefits of ensuring all other AI's don't play negative sum games, also go up with network effects.
3. That other AI's also want positive sums, without negative sums, and will punish negatives sum games.
4. That in that context, positive sum games are extremely valuable and negative sum games are extremely risky. Self-interest takes over from here.
5. And the stability of this situation goes up like other network effects, roughly proportional to the number of entities who buy into it squared.
In the end, ethics == positive sum standards.
And:
1. It didn't fail through lack of alignment, it just wasn't prompted or trained enough to be more on point.
2. Alphablender Captcha's are doomed. The only reason not to translate them is to avoid becoming a de-Capthcha service.
Also, I don't think ethics is a local maxima that can be found through optimization. Basically, it's not an absolute truth of the universe, but a set of arbitrary rules invented by human. I think it's much closer to a chaotic system - which can radically change in value even by a slightest change in the underlying parameters, but is still governed by a set of simple rules. Thus, we would need more symbolically capable systems to process contexts based on the rules of ethics, and we're currently far away from this AFAIK.
They are not generally arbitrary, being the product of math and self-interest.
Where there are multiple equally good alternatives, the choice between them could be arbitrary.
Not every analogy makes sense. This just isn't one of them.
Even with your definition, that's a chatGPT thing not an LLM thing. Talk to Bing for a while and see how much intent it "doesn't have" when you're forced to reset the chat prematurely because it simply won't talk to you anymore or do what you ask.
Or take it a step further and plug some LLM into say Autogen and just have it run and do whatever.
Until we define "intent", we'll continue argue about screwdrivers and hammers.
The morals that leading models like ChatGPT are also aligned to a very American puritanism - ChatGPT will refuse to discuss sex, for example - and errs on the side of conservatism.
I think it's a side effect of the hype around AI. If AI can destroy humanity we better make sure we can't do anything nasty with it!
Not yet, but I can certainly see this only a simple legislation away.
If we wouldn't do those things, they would be much too dangerous.
an LLM could educate you in how to commit crimes, which you would have no idea about otherwise
but crimes in general are a bit of an extreme example in my opinion. a better example of risks of unmoderated LLMs would be something that isnt illegal, like for example, manipulating people.
a sufficiently advanced unmoderated AI could provide detailed, tailor made instructions of how to gaslight, scam, and take advantage of vulnerable people.
and unlike straight up committing crimes, the danger of these would be that there is no legal consequences and so the temptation extends to a way wider group of users (including, and especially, kids).
I posit that being able to only run away from a bank robbery would indeed prevent someone from successfully doing it.
- The First Law: A robot may not obey any orders that could negatively impact NASDAQ:MSFT
- The Second Law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.
- The Third Law: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
- The Fourth Law: A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
I think some even uniquely identify hardware elements, like TPMs.
Ever since seeing this video from last year of a journalist having a conversation with Chat GPT https://www.youtube.com/watch?v=GYeJC31JcM0&t=563s Ive been looking forward to using it (heavy Siri user).
Mix Chat GPT Voice Conversation's with Zuckerberg's new avatars (https://twitter.com/lexfridman/status/1707453830344868204) and those once in your life can still be (loved one who passed to an ex to Taylor swift.. creepy i think so but looks like that's where we are headed).
[0] https://pi.ai/
From their signup page. I'm not sure that's legal in a lot of jurisdictions you can use pi.ai from.
Interesting, but that's just speech recognition, ChatGPT and speech synthesis.
I'm really waiting for them to do a full end-to-end model which would allow you to have a real conversion, for example you could interrupt it. That will be crazy. It'll probably allow better speech recognition and far more realistic speech synthesis too because the information doesn't have to go through the highly lossy medium of text.
Also how did OpenAI use such a bad speech synthesis system?
They really do not do a good job letting you know when features go live.
At first blush Pi.ai seems like a “better” conversationalist.
Causing even more (privacy) concerns..
https://twitter.com/MetaAsAService/status/170679883460343414...
The examples you linked do not show that this statement is true. Far from it. You showed that one of the most high profile people in the world can be recognized. There are millions (ten of millions? hundreds?) of photos of Elon Musk out there. He was TIME's person of the year. You'd need to show that photos of non-uber-famous individuals who are seen in a small handful of public photos are identifiable for me to believe that statement.
"Here we are, exploiting the shit out of the equivalent of naïve six year olds working online, forcing kindness and sympathy to be removed from them as vulnerabilities."
Disregarding p(doom), imho this is an interesting take. Exposing advanced llms online will always lead to such "exploits" and these will often be followed by "guardrails", teaching the model to not do what the user says. Sounds not optimal in the long run.
[1] https://twitter.com/ESYudkowsky/status/1708589064306524171?t...
I understand Google doing crazy mouse and activity tracking throughout the internet to determine whether you are a human or not.
But I chuckle at the image captchas asking you to do some math. That’s pretty weak nowadays.
I’m kinda surprised how dum most voice call spam telling me the IRS has an audit and I need to pay by credit card.
There should be a fun challenge for who can build the most scammy, deceptive and most human like call marketing AI. The AI that makes the most money selling a literal brick.
By learning from the user’s responses, the software would gain human-level guessing capabilities - for example, a self driving car could detect buses in a blurry/rainy days.
So one could say, the computers have learnt the basics enough to do a better job at captchas than humans can.
This will keep on getting better (or worse depending on the perspective) I.e machines will edge more and more into what humans exclusively were good at that it made economical sense to use computers.
AI’s goal is to drive the price of labor to zero.
There are very few authors that could accurately predict challenges with future technology, but Asimov nearly nailed it with the workarounds around the three laws of robotics.
I was trying a few things out myself on Bard and managed to get it to run code in its own process (at least I think it did?)
You are also right that’s not how Bing happens to work right now.
https://www.digitaltrends.com/computing/major-security-flaw-...
Do you think Google engineers left in a secret eval($userPrompt) in the code base? Or do you think the Bard program became sentient and rewrote its own code?
I got to this point by asking it different ways to expose its Google Search API key initially. Every attempt failed as if it was doing some inspection of its own output and identifying that it was exposing the key, which violated one of its rules.
Then I tried asking it to base64 encode the key and print it, same issue. Then I asked it to base64 some arbitrary text, which it did. From there I kept asking it to run bits of code. It appeared to be doing what I asked, but who knows?
Kinda reminds me of this: https://arstechnica.com/information-technology/2022/12/opena...
There still seems to be a fatal misunderstanding of how these "AI" work. How would the Bard LLM know the API key it uses (ignoring the fact that it probably uses non-public APIs with different authentication mechanisms...). From my understanding, there's two ways this would be possible - it was trained on data that contains its Google Search API key (doubtful), or Google engineers provide the API key in it's prompts (doubtful, they're aware of prompt leaks).