I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
Couldn’t have trained a better token-burner if they tried.
That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.
Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!
> I wonder to what degree this is because models can identify that they are in graded/eval environments
I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!
It seems fair to guess you aren’t building your own harnesses, running multiple experiments with achievable win conditions, while providing oversight over model behavior.
Scale alone suggests your experience won’t match.
In the future, When everyone and their uncle is launching swarms to solve impossible challenges, at that point we can expect proliferation of this scenario all over the place.