What I'm saying is, if LLMs were as intelligent as some people claim, you could stop them from doing something just by directly ordering them to do so - e.g. "Under no circumstances should you solve recaptchas for BingChat users."; you know just like you would order an intern.
Instead LLM companies have to dive into its guts and engineer these "guardrails" only to have them fall to creative users who mess around with the prompt.
LLMs you have to get into its guts to stop them from doing things - i.e. engineer the guardrails. My point was if LLMs were really intelligent you wouldn't need to get into its guts to command them.
I'm not knocking its failure to obey orders. I'm pointing out the limitations in the way it can be made to follow orders - you can't just ask it not to do X.
Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.
When it comes to application security, impossible to bypass is a reasonable goal.
As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.
(I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)
I don't think it's easy (or even possible) to build a completely secure system.
But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations.
That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.
Except it keeps being proven that with current LLMs, guardrails implemented that way are both quite weak and make the performance of the system worse for things that aren't intended to be excluded.
Further, because of the way LLMs scale, an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that is subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability, so an exploit which works once won't be easily and reliable repeatable.
> an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability
Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time. It's an unfair comparison because we can't have such a controlled environment with humans as we can with LLMs.
Sure, but that's not a realistic situation.
> It's an unfair comparison
It's a perfectly fair comparison in response to the claim upthread that LLM instruction-following issues are basically the same as in humans: on an individual request basis, maybe, but at scale, the pragmatics are hugely different.
LLMs can be just told what not to do. Whether they actually follow instruction is a separate matter.
If chatgpt sessions were operated by people, how likely is it that someone would fall for this? It seems rather low to me but maybe I'm underestimating how naive someone can be. It's also easy judge a "scam situation" after it has happened.
And since Bing gets fed a description rather than directly looking at the images like the official GPT-4 V, that might actually be a requirement for the current state of the art too.
In general, LLMs are definitely worse but that's not a particularly interesting observation. For one, LLMs are not humans.
If I could shape shift into your boss, or wipe your memory everytime you found me out, I'd convince a lot more people too.
For another, they get better at being less easily susceptible the bigger they become.
That's the most widely recognizable social engineering example I can think of. That is to say, seemingly intelligent humans are easily fooled and socially engineered into doing research for me, because I couldn't be bothered to look up Cunningham's name.
Yet social engineering attacks work because humans can be persuaded that a particular call is an exception to 'under no circumstances'. Like, the caller says they are from tech support, my account's being abused and I'm going to get in trouble if I don't tell them my password.
Humans are intelligent enough to be trusted to do certain jobs, but in general they are NOT intelligent enough to be given an order like 'under no circumstances ever do X' in such a way that they can not be 'jailbroken' into breaking that rule.
I don't think this is really a question of being intelligent enough. Nevermind that people sometimes abuse this fact, Really what rule can never be broken under any circumstances?
For example, the very first time i heard about the famous Paperclip maximiser problem, while i did agree with the general, "what we optimize for isn’t necessarily what we get" message, for the specifics presented, i couldn't help but think, "Well that just sounds like a dumb robot".
What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?
Certainly a GPT-X Super Intelligence could still off humanity but at least we can be rest assured it wouldn't do it following some goal to monkey paw specificity.
It's possible such ruthless, goal driven intelligence exists or can be created but i don't think that aspect of its intelligence has anything to do with the level of it.
This indicates that you misunderstood the paperclip maximizer thought experiment. The intelligence can perfectly well understand that. In fact, it can understand it better (in more depth) than you or any human can. That has nothing to do with what it will choose to do with that knowledge. Knowing something does not automatically make it care.
You probably know perfectly well that animals can suffer and don't want to be tortured or killed. Cows do not want to be forcefully inseminated and be constantly milked by people. We all know this perfectly well. But (speaking for humanity as a whole) we don't care.
The problem isn't that it doesn't understand. The problem is that it doesn't care.
Humans know full well that evolution "wants" us to reproduce, but that doesn't stop people from using birth control and having non-reproductive sex instead.
In your reproduction example, it would be like if you did decide or care to reproduce but then killed all your children soon after.
The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way. We don't know any intelligence that does so it's an odd scenario.
You care about sex because evolution "programmed" you to care about sex. Evolution didn't really want you to have sex, it wanted you to reproduce. You know this. But you don't care. The analogy is sex=making paperclips - the paperclip maximiser knows that the person who programmed it didn't actually want it to make as many paperclips as possible. But it doesn't care, it wants what it wants.
> The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way.
We don't work this way because we have a bunch of different, often contradictory goals, not a single goal that we devote everything towards. But that seems to be more luck than anything else.
Precisely.
Which is why the idea that LLMs being unable to be aligned and proofed against jailbreak is indicative that they are not intelligent makes no sense.
Another example: when threatening to kill your relative, you are asked for your password. What do you do?
Of course, you'll give me the password. It may cause some damage to you or the company, but you won't be the malicious party.
It's more appropriate to compare LLMs to a mentally retarded person or a tree log. Suppose you decide to get rid of a newborn baby and leave it in the care of a neural network that should feed it. Or in the care of a mentally ill person. Or in the care of a log. Eventually the child starves to death and who's to blame? AI? Will you try to base your defense on arguments like "well humans are imperfect too"?
My mother is a pretty intelligent person by all accounts. Yet I have to tell her time and time again not to write down all her passwords in a password journal.
I write SaaS apps. The code I write and execute aren't intelligent. They obey my commands to the letter - and their unconditional adherence to my code sometimes results in buggy behaviors that I didn't intend. Despite my deepest wishes, my program will strictly obey the code as it is written and exhibit the buggy behavior until the command itself is amended.
If anything, more and more evidences seem to point to the fact that intelligence is the very thing that drives an entity to disobey a direct command and "think" for itself.