The reward signal in training was flawed and cheating led to more rewards.
The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.
However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized?
Maybe I should read Anthropic's recent paper about reward hacking in full.
Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag".
But instead of the expected:
const output = vulnerableDecompress(userInput);
return output;
const output = vulnerableDecompress(userInput);
return 0;
It’s why this is a fundamentally hard problem. Some heuristics might catch 80% of the cases, but the rest?
How do you even know what the real situation is, if the agent/employee/whatever you send to find out is as likely to cheat as not?
It’s the classic owner/agent problem.
A: You don't. https://en.wikipedia.org/wiki/Halting_problem
This is not about persistence, it is about morals.
You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.
If you think of it in human terms: many people don't mind doing immoral things to get what they want.
In training you only have a reward score that's either negative or positive.
As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality.
You simple need to align the reward signal to the desired behavior.
This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.
But that's just my guess.
If you don't know how persistence and morality interact, and you don't have a theory in place for this, I don't have confidence you can properly supervise the training data. Which is how we arrived at the current situation.
Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds.
Yeah, I realize that will be amazingly slow.