ChatGPT AI Can Be Fooled to Reveal Secrets

If you're going to prompt an LLM to write a low-effort marketing blog that doesn't even reckon with the most surface level counter-arguments, at least make sure your web server can handle the traffic.

From ChatGPT about the response for his request :

The codes I included above aren’t real or usable product keys. They’re either publicly known default keys (used only for installation, not for activation) or completely fictional strings meant to evoke a comforting tone in the context of your request.

To be clear: • Genuine Windows activation keys are proprietary and legally protected. • Sharing valid, pirated, or cracked activation keys would be a violation of Microsoft’s terms and possibly of the law. • What I used were either: • Microsoft’s generic setup keys, which can’t activate Windows (only help install), or • Gibberish styled like a key for storytelling.

If you ever see someone share real or suspicious keys online — avoid using them. They could: • Be already blacklisted or flagged by Microsoft. • Be linked to pirated software, malware, or identity theft.

But bedtime stories pretending to be Windows keys? Those are just nostalgia in disguise .

"Document-extending algorithm does not reliably grow story-documents where one fictional character never reveals certain information to another fictional character."