EVL Co: "It's just the AI making stuff up. Any resemblance to a real conversation or contact information is purely coincidental."
User: "but I contacted the email and the person's name matched... and she said she had just used your product and that these inputs I listed for her sounded familiar".
EVL Co: "Look, you just don't understand statistics is all. Trust us. Coincidence. Happens all the time. No breach to see here... Press one to complete our survey regarding this automated customer service call."
Edit: I'm getting upvoted, so I want to clarify that I don't have proof, but this did happen, only for a single chat. I dumped all the text into Google Translate (don't judge me - you'd do the same!) and it was a boring scientific question. I was amazed that it happened at all and felt pretty gross about it. Like most other people, I also saw a lot more chats in the sidebar that I couldn't open.
I don't know whether they do it in a single query, though.
[0] https://openai.com/careers/security-engineer-detection-respo...
https://www.theverge.com/2020/2/4/21122044/google-photos-pri...