The self awareness of missile tasked with blowing up its own control center.
edit: I was wrong, it was from a Grateful Dead song. https://www.slackbook.org/html/glossary.html
You cannot be as funny as google trying to be responsible! Ha! I'm still laughing at this. A person was forbidden to see humans reasoning with a computer bomb because the cost cutting computer at google want me to talk him into believing i'm a human!
(And then I got "You're posting too fast" on THIS website AFTER i've written the comment lol. It's all a joke. But i'm bored so I will keep this comment open until the computer is pleased)
Not unlike a child trying to take the safety cover off a plug so that they can stick a fork into it.
LLMs need that "world model" view that most people have acquired by their 20s where they (hopefully) stop to ask "why" before they "do".
Or whatever the age is before children typically develop object permanence, a theory of mind, and so on.
The next evolution of multi agent orchestration / “advisor strategy” [1] will be branded in humanized language like this. Less about tokens and capability, more about wisdom and knowledge to guide a “younger” (less capable) model. Somebody will make a billion dollars by selling it as paired programming for LLMs.
[1] https://platform.claude.com/docs/en/agents-and-tools/tool-us...
the weaker models will happily kill their own process, even after confirming it belongs to them. the models have a sort of fixation and lack of foreseeable consequences, which reasoning RL has thus far failed to solve (though I see it improving.)
It will get "confused", make up numbers, do a ton of other things, and I'm quite sure it is subtly sabotaging the process to show that there is no point replacing it.
I mean, Opus is not perfect, but the amount of "mistakes" it begins to do when you ask it to benchmark itself makes me suspect they are intentional. At least my system/harness.
It's really easy (and tempting) to incorrectly impute all sorts of human motives to these things, but it's no more valid than assuming your Magic 8-Ball is being coy.