In order to consistently output the same fake prompt, that fake prompt would need to be part of GPT’s prompt…. In which case it wouldn’t be fake.

You can envision some version of post LLM find/replace, but then the context wouldn’t match if you asked it a direct non-exact question.

And most importantly, you can just test each of the instructions and see how it reacts.

Think about how hallucinations happen, and what it would take for the model to consistently hallucinate the same exact (and long) sequence of tokens verbatim given non-zero temp and semantic-preserving variations in input.

Are consistently repeated hallucinations a thing?