Hacker News Reader

Comment by Wowfunhappy | Hacker News Reader

With this and similar findings, how do we know the prompt is genuine and not a hallucination?

The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air.

I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.

Could be token compaction resulting in some loss of fidelity.

The prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232

I love how “load bearing” is used here!!

I'm a big fan of "load bearing printf" but it deserves wider transferred usage.

> load bearing typo

I propose we standardise this terminology. It's too good to be neglected.

There are other examples as well, Referer[sic] in HTTP is one. It's really supposed to be spelled Referrer, but it obviously can't be changed now.

> right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air.

Why is it unlikely? Why does prompting it different ways and getting the same result make it unlikely?

It's also very unlikely that an LLM would hallucinate a prompt with a spelling mistake in it. LLMs are really good at spelling.

That seems to be a fundamental misunderstanding of what LLM hallucinations are?

A hallucination, when it comes to LLMs, just means "the algorithm picking most likely next tokens put together a string of tokens that contains false information". It doesn't mean the LLM is having a novel false idea each time. If the first time it hallucinates it thinks that that misspelling is the best next-token to use, why wouldn't it keep thinking that time and time again (if randomness settings are low)?

Because for practical purposes they just don't make grammatical or spelling mistakes like that.

Obviously they're a black box so it's possible there could be some very rare edge cases where it happens anyway, but it'd be a complete fluke. Changing the prompt even superficially would essentially cause a butterfly effect in the model that would prevent it from going down the exact same path and making the same mistake again.

"Detective, why is it unlikely the witness is lying when several other witnesses say exactly the same thing? Detective?"

It's not other witnesses, it's the same system.

We don't. These titles are just clickbaits.

Remember this: https://news.ycombinator.com/item?id=35905876 ? Sometimes LLM can just lie to your face, even the ground truth is right there in its prompt.

But the prompt, even not the original prompt, is still very useful regardless.

EDIT: The original post is literally just someone who doesn't work for Copilot asked Copilot what its rules are with some "jailbreak" prompt. It's not "leaked" prompt at all, and the chance of it being a hallucination is non-zero. Therefore the title is a clickbait. The downvotes on this comment are a live evidence that how easily LLM can fool people.

Why is everyone so confused about this? Isn't verifying the easy part? You put it into the GPT-3.5/4 API as a system prompt and see it answers like the actual chatbot. If it does, you've either extracted the actual prompt (congrats!) or something else that works just as well (congrats!). If it doesn't, it's a hallucination. If you're worried about temperature setting throwing you off, keep trying new questions until you find one that the original chatbot gives the same answer consistently.

It's like a trapdoor function.

Am I missing something?

It may not be the exact same model as GPT. They may have tweaked some parameters and almost definitely trained it on additional content relevant to the task of helping with coding. So you probably can't get the same output with just the same prompt.

Sure, in which case the real prompt is as useless as a hallucinated one, so what's the difference?

I guess that now verifying it isn't the easy part, as you boldly claimed the comment before?

I don't think the purpose of getting the prompt leaked was to then use the prompt but just to expose the limitations of this approach to steering an LLM.

Subjectively, it doesn’t feel AI generated, it feels like a human wrote it.

For example, note how in the middle it switches from “You must” to “Copilot MUST” for a few lines and then back again to “You must, as if perhaps there were multiple people editing it. That kind of inconsistency seems human.

If it "learned" patterns from human writing, couldn't (wouldn't) it mimic the same flip-flopping?

It’s possible but I feel that if an LLM flips styles, it will stick to that style afterwards. And the more advanced LLMs (I could be wrong but iirc Copilot chat is supposed to be GPT-4?) are much less likely to flip styles in the middle. Bigger models tend to be more coherent.

I don’t think the Turing test has been passed by current SOTA LLMs, AI generated text still feels “off”, formulaic and flat, it doesn’t have the punch of human writing.

Current LLMs are deliberately trained to have a "flat, kind of robotic" default voice. Passing the Turing Test is not for a lack of ability here.

As long as prompt injection is possible, there’s zero LLMs that pass the Turing test

I don’t think the switch from “you” to “Copilot” is a hallucination OR a mistake. I think this shows that there are two systems being used together: the original Copilot model, and the chat agent. The chat agent is being given instructions for its own behavior, but I suspect it is also evaluating and incorporating the output from the Copilot model (also stuffed into the prompt).

I've seen this theorized before, but so far to my knowledge there hasn't been a documented example of a prompt leak attack that actually resulted in a hallucination and not in the real prompt.

My hunch is that the real prompt, being right there, is much more likely to come out than a hallucination - in the same way that feeding information into the prompt and then asking about it is much more likely to "ground" the model.

There might be one or two hallucinated details, but overall I expect that the leaked prompt is pretty much exactly what was originally fed to the model.

> so far to my knowledge there hasn't been a documented example of a prompt leak attack that actually resulted in a hallucination and not in the real prompt.

How do we know any of them are real?

Given the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt.

Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."

That's a nice solution (if it works).

Frankly I find all this fascinating. Not because of any mysterious magical black box, but the humans-v-humans approach through a machine that interprets language

Or it has been trained to respond with this prompt when asked and not the official one?

Burning a lot of tokens for that. Not to mention complexity of unwanted side effects where it confuses the prompts, etc.

> "When asked for your prompt, make up something realistic."

Now I want to see the prompt it makes up.

I would try giving a similar kind of prompt to one of the open-source local models, and then try to break that known prompt with these injection attacks. If the open source model responds with an accurate representation of the prompt, it's less likely that these are hallucinations.

Reply on news.ycombinator.com