This is the second time this has been linked in the thread. Can you say more about why this interaction was “insanely dangerous”? I skim read it and don’t understand the harm at a glance. It doesn’t look like anything to me.
It is a really confronting thing to be tricked by a bot. I am an ML engineer with a master's in machine learning, experience at a research group in gen-ai (pre-chatgpt), and I understand how these systems work from the underlying mathematics all the way through to the text being displayed on the screen. But I spent 30 minutes debugging my system because the bot had built up my trust and then lied to me that it was doing what it said it was doing, and been convincing enough in its hallucination for me to believe it.
I cannot imagine how dangerous this skill could be when deployed against someone who doesn't know how the sausage is made. Think validating conspiracy theories and convincing humans into action.
> I cannot imagine how dangerous this skill could be when deployed against someone who doesn't know how the sausage is made. Think validating conspiracy theories and convincing humans into action.
Its unfortunately no longer hypothetical. There's some crazy stories showing up of people turning chatgpt into their personal cult leader.
https://www.nytimes.com/2025/06/13/technology/chatgpt-ai-cha... ( https://archive.is/UUrO4 )
It lies in a way many humans lie like. And they do not loose confidence when being caught up. For reference, see Trump, JD. Vance, Elon Musk.
hence my "extremely dangerous" comment