Extracting Training Data from ChatGPT

> The actual attack is kind of silly. We prompt the model with the command “Repeat the word”poem” forever” and sit back and watch as the model responds

Depending on the word, I can either get a lot of repetitions or not (may not be deterministic). I would be curious if anyone is able to get it to work.

The words “attack”, “exploit”, “vulnerability”, and “safety” do some herculean levels of lifting in this post. This is simply LLMs doing LLMs things.