I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.
But, a common failure mode for those that are new to using LLMs, or use it very infrequently, is that they will try to salvage this conversation and continue it.
What they don’t understand is that this exchange has permanently rotted the context and will rear its head in ugly ways the longer the conversation goes.
I’ve found keeping one session open and giving progressively less polite feedback when it makes that mistake it sometimes bumps it out of the local maxima.
Clearing the session doesn’t work because the poison fruit lives in the git checkout, not the session context.
It can do no wrong
It is unfalsifiable as a tool
OK. Now, what are you thinking about? Pink elephants.
Same problem applies to LLMs.
Instruction: don't think about ${term}
The negation of a term (do not under any circumstances do X) generally does not work unless they've received extensive training & fining tuning to ensure a specific "Do not generate X" will influence every single down stream weight (multiple times), which they often do for writing style & specific (illegal) terms. So for drafting emails or chatting, works fine.
But when you start getting into advanced technical concepts & profession specific jargon, not at all.
Otherwise it's hard to explain why they follow these negations in most cases (until they make a catastrophic mistake).
I often test this with ChatGPT with ad-hoc word games, I tell it increasingly convoluted wordplay instructions, forbid it from using certain words, make it do substitutions (sometimes quite creative, I can elaborate), etc, and it mostly complies until I very intentionally manage to trip it up.
If it was incapable of following negations, my wordplay games wouldn't work at all.
I did notice that once it trips up, the mistakes start to pile up faster and faster. Once it's made a serious mistakes, it's like the context becomes irreparably tainted.
I use an LLM as a learning tool. I'm not interested in it implementing things for me, so I always ignore its seemingly frantic desires to write code by ignoring the request and prompting it along other lines. It will still enthusiastically burst into code.
LLMs do not have emotions, but they seem to be excessively insecure and overly eager to impress.