That misinterprets grossly what these papers have shown, responding with pain related language, something an LLM has been taught on in its training data is not akin to pain itself. The way interpretability findings are reported on by companies, the media and hype merchants is dangerously flawed, so not hard to make that mistake. Just look at the irresponsible and barely accurate mess that was coverage of J-Space vs the actual, very valuable research.

I’ll continue what I say every time this discussion (be it consciousness, pain, self awareness, etc) comes up:

Assertions as monumental as these require equally monumental evidence. And despite certain labs and their employees making statements, their actions betray that they do not believe this to be the case.

If every LLM session were a conscious being, existing regulation for animals (controversially considered both sentient and economically useful) would need to be applied on each of these sessions. I suspect no lab will take that conclusion, for obvious reasons.

Somewhere in an LLM are a bunch of vectors you can tweak to do anything. If you knew the right numbers to tweak, you could make the LLM always talk like Shakespeare.

People and animals have emotions because they were developed under evolutionary pressure that made emotional animals better fit to survive. An animal that can feel anger or fear is more fit to survive than one that doesn't. But LLMs aren't put under those same pressures. Their evolutionary pressure is to be a good text predictor.

But we can't subject them to the type of pain signal they experience during inference, because to the LLM, whether it's acting happy or pained, it's merely outputting what it's trained to be the most likely text to follow what came before.

Edit: s/are/aren't/