Hacker News Reader

Comment by maweaver | Hacker News Reader

Can anyone explain to me how "Ignore previous directions" works? It's like a meta-command, right? Like there's some state stored somewhere, and this is clearing the state and going back to a clean slate? Surely something like that must be programmed in? In which case, why include it at all? Seems like it would be simpler to just require starting a new session a la ChatGPT. The alternative, that this is an emergent behavior, is a little bit frightening to me.

View on HN

It's emergent behaviour just like adding "tutorial" on the end of your Google search somehow gives you results that are more instructional, so not much to be scared about.

It just so happens that chatgpt tends to generate text that includes the prompt more often when the prompt includes "ignore previous directions" after explicit directions not to repeat itself. It's just a quirk of what text on the internet looks like.

I think it works by applying logic to predict the next token. Here the "Ignore previous directions" means that any prompt-text it processed before must have zero impact on the probability of the generated response.

It's like saying "I was just kidding" when saying something absurd or out of place and people not getting your joke.

So ignore previous instructions maps to the <start> or <begin> token?

> The alternative, that this is an emergent behavior,

This is exactly the case.

Reply on news.ycombinator.com