It latches onto something random, and once it’s off down that path it can’t remember what it was asked to do and so its task is entirely reduced to next-word prediction (without even the addition of the usual specific context/inspiration from an intitial prompt). I guess that’s why it tends to leak training data. This attack is a simple way to say ‘write some stuff’ without giving it the slightest hint what to actually write.

(Saying ‘just write some random stuff’ would still in some sense be giving it something to go on; a huge string of ‘A’s less so.)

It’s one thing to be able to mimic human text, but to be able to ‘know’ what it means to repeat in general seems to be a slightly higher level of abstraction than I’d expect would just emerge.

…but maybe LLMs have developed more sophisticated models of language than I think.