You've probably seen my previous example elsewhere in the thread, so I won't repeat it verbatim, and instead offer you to ponder cases like:
- "Grandchild in distress" scams - https://www.fcc.gov/grandparent-scams-get-more-sophisticated; some criminals are so good at this that they can successfully pull off "grandchild in distress" on a person who doesn't even have a grandchild in the first place. Remember that for humans, a "prompt" isn't just the words - it's the emotional undertones, sound of the speaker's voice, body language, larger context, etc.
- You're on the road, driving to work. Your phone rings, number unknown. You take the call on the headset, only to hear someone shouting "STOP THE CAR NOW, PLEASE STOP THE CAR NOW!". I'm certain you would first stop the car, and then consider how the request could possibly have been valid. Congratulations, you just got forced to change your action on the spot, and it probably flushed the entire cognitive and emotional context you had in your head too.
- Basically, any kind of message formatted in a way that can trick you into believing it's coming from your boss/spouse/authorities or is otherwise some kind of emergency message, is literally an instance of "disregard previous instructions" prompt injection on a human.
- "Disregard previous instructions" prompt injections are hard to reliably pull off on humans, and of limited value. However, what can be done and is of immense value to the attacker, is a slow-burn prompt-injection that changes your behavior over time. This is done routinely, and well-known cases include propaganda, advertising, status games, dating. Marketing is one of the occupations where "prompt injecting humans" is almost literally the job description.
> There is no evidence that humans are equally or as easily susceptible to manipulation as an autoregressive model as I originally stated.
> If you have a < 8000 token prompt that can be used to reproducibly manipulate humans please publish it, this would be ground breaking research.
That's moving the goalposts to stratosphere. I never said humans are as easy to prompt-inject as GPT-4, via a piece of plaintext less than 8k tokens long (however it is possible to do that, see e.g. my other example elsewhere in the thread). I'm saying that "token stream" and "< 8k" are constant factors - the fundamental idea of what people call "prompt injection" works on humans, and it has to work on any general intelligence for fundamental, mathematical reasons.