If you’re showing off a novel example disregard the question about. This example highlights how LLMs will treat any question with an attempt at a truthful answer even if they don’t know the truth themselves. I’m not sure if RLHF alone will be able to solve for this failure mode.
It's been weird to me how many people in this forum sincerely seem to believe that this is more than a language model. It's a really good language model, and has the potential to be useful for a lot of things, but it's not a repository of facts*.
* EDIT: It has a lot of facts embedded but should never be trusted to reliably tell the truth. It is first and foremost a model of human language, and lies are expressed in language, too.
For example: you talk about philosophers. Then you refer to them as "they". From then on, it will remember the context whenever you refer to philosophers as "they". When pressed, it will repeat your mantra of "I am just a language model, I can't possibly remember context" while it clearly does, either by design or emergence.
It's good to think about these models as eager cosplayers of any role you set up in your prompt. In this case, it sees its own past replies, so it cosplays an AI that gave these answers before. You could edit its past replies and override its "personality" this way.
Basically it's a contextual trick that exploits our pattern recognition pareidolia, like faces in clouds or horoscopes or cold reads.
It's not its code. It "cold read" what you wanted from the questioning itself.
Cold readings commonly employ high-probability guesses, quickly picking up on signals as to whether their guesses are in the right direction or not, then emphasizing and reinforcing chance connections and quickly moving on from missed guesses. Psychologists believe that this appears to work because of the Forer effect and due to confirmation biases within people.