Hacker News Reader

Comment by nkrisc | Hacker News Reader

Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it?

We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.

View on HN

But humans are very very good at this specific problem.

Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do."

Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc.

Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!"

The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.

I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.

And yet a prompt injection attack could never work on a customer service assistant.

Sure they can be tricked, but prompt injection is about conflating trusted instructions and user input. There is no chance you'll convince a CS rep you're secretly the CEO over a web chat UI.

I imagine a few customer support reps were tricked by support tickets that were filed by attackers. “This is a test from our Security Team, open this link to confirm you have passed it”

Yes, humans are good at it. Is your language model?

> But humans are very very good at this specific problem.

Humans are vulnerable to “prompt injection”, but not identical forms to each other because humans don't have identical “training data” and “hidden prompts” to each other the way GPT-4 sessions via identical frontends do. Also, the social consequences for unsuccessful, and after-the-fact identified successful, prompt injection attacks on other humans are often much more severe than for those on GPT instances.

Reply on news.ycombinator.com