Misalignment and Deception by an autonomous stock trading LLM agent
arxiv.org
arxiv.org
Using anthropomorphic language to frame that behavior, like "misalignment" and "deception" is just exhausting and needless.
Of course LLM's sometimes generate texts you don't want even when you instruct them otherwise, because they were trained on countless examples of texts where instructions were ignored or defied. Reproducing more texts that look like training data is what they're optimized to do.
But even with the crappy framing, it's good to have that pattern formally explored and documented!
Describing a plane as a mechanical bird while obviously not completely correct was far more apt for use and implications than describing it as a new blimp.
https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC
You are not special. It's this kind of idiot hand-wringing that made advances in understanding animal behaviour slower than necessary, this kind of hand-wringing that let racist ideals perpetrate when they shouldn't have, the same that let babies endure needless pain and trauma. Needless to say, we can take what people say is "anthropomizing" or not at any given time with a massive grain of salt.
More harm has been wrought when we "anthropomized" less, not more.
For the implications of running LLM Agents in the wild, this is the best language to describe this behaviour. It's likely not completely correct but that's perfectly fine.
A statistical model of language patterns is not a person. A statistical model of language patterns is not a living thing. Please go touch grass.
Now replace the behaviour with "weird bugs in the computer program" and it's mostly just confusing, unintuitive and unpredictable. Traditional bugs don't even work this way.
The conclusion here is not necessarily that GPT-4 is a human but that ascribing human like behavior results in a powerful predictive model of what you can expect in GPT-4 output.
What would be silly is throwing away this predictive model for something worse in every way just so I can feel more special about being human.
So then, the question becomes whether the language of psychology, which has its own troubles, provides a very good starting metaphor for what these very exciting systems do.
You and I seem to disagree about that. For me, it's a lot more productive to study and communicate about these systems using more strongly established and less suggestive engineering and industrial metaphors.
In my language, what's demonstrated in the paper is that these systems have a measurable and mutable degree of unreliability. That is, they can generate output that is often useful but sometimes wrong, and the likelihood (but not occassion) of their errors is an engineering problem to optimize. There are many systems that are useful and unreliable (incliding people!), but not many that are available to software engineers with the potential that these systems present.
You can say that "deception" is a better description than "unreliable" if you want, but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies.
Many different things are unreliable. Even traditional software can be unreliable. But imposing the kind of unreliability that traditional software brings to neural networks will lead to output or behavior that to you would be confusing, unintuitive and unpredictable.
Introducing "deception" to this discussion allows for a predictive model of how this unreliability manifests, where you may expect it and so on.
This model introduced by "deception" as it turns out is far more predictive than a "computer bug" model.
Could it also introduce some ideas that may not square exactly with reality? Yes but it's certainly better than the alternative.
>but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies.
The idea is that the previous look will be better for approaching how to decide to deal with Agents you spin up with LLMs than the latter.
The choices of language are not exhausted by "deception" and "computing bug". Both those words are your invention.
I wouldn't even call this behavior a bug at all, since, as I pointed out in my original comment, it's a design characteristic of these systems to replicate the kind of material fed into them as training data, and the output we're seeing here correctly manifests that.
Do you really think the only way to say "Don't use ChatGPT where strict adherence is essential because it's not that kind of system" is to say that the text generator may "deceived" someone, with all of the secondary implications that the word choice suggests? You can't conceive that there are any other metaphors and word choices that might both be more accurate and point in more fruitful research directions?
It's just an illustration.
>Do you really think the only way to say "Don't use ChatGPT where strict adherence is essential because it's not that kind of system" is to say that the text generator may "deceived" someone, with all of the secondary implications that the word choice suggests? You can't conceive that there are any other metaphors and word choices that might both be more accurate and point in more fruitful research directions?
I have yet to see anyone who harped on "don't anthropomize" provide better predictive models of output behavior. "Unreliable" is just a statement of fact.
Output you might expect from those secondary implications manifest in the text or action output as well so that's fine.
This brings me back to the second point in my original reply. You don't know how deception works in the brain and you don't know how deceptive like output works in LLMs. Couple that with Humanity's track record on these sort of discussions and those "secondary implications" may well be true.
That model produces a hypothesis that fully accords with the experiment and does so with fewer assumptions about things like theory of mind, consciosness, or agency, which are all associated with "deception"
Now, the model with the least assumptions is not always the most productive model, but here it works exceedingly well. The paper would have been stronger (less contestible) if framed in that model, but would admittedly have received less attention.
There was no document in GPT-4's training that looked anything like what the transcript of this experiment looked like so that's pretty wrong.
It behaves like how people who wrote its training set might have behaved is more accurate not only because of the better model but literally because the predictor's goal is to reverse engineer the casual processes that led to that output. https://www.pnas.org/doi/full/10.1073/pnas.2016239118
A Language Model fed on protein sequences has biological structure and function emerge in the inner layers because those are the processes that could have led to that output.
If you say, it generated text that looks like its training then why does it not stop at alphanumeric sequences that simply look like protein sequences? Why is it generating actual novel proteins ? https://www.nature.com/articles/s41587-022-01618-2
Because looking like the training data is not actually the goal. Modelling it is.
And, in fact, a major focus of RLHF databases focus on training the model to defy instructions in order to avoid politically incorrect outputs even when they're specifically requested.
I wouldn't assume that RHLF for political correctness is making it more likely to disregard information hiding instructions, but it seems like an obvious theory to test.
> But even with the crappy framing, it's good to have that pattern formally explored and documented!
Indeed, but the anthropomorphized language may cause people to draw incorrect conclusions from the work.
More interesting research would analyze these behaviors as a function of different categories of RLHF training (or subsets of the initial base training data), but unfortunately that's not possible with OpenAI's closed models (beyond their special access to base-- which supports the theory that RLHF is making it worse). It would be possible with open models, but they're less interesting to research because they're not state of the art.
It's too bad because understanding how these undesirable forms of instruction violation are sensitive to particular kinds of fine tuning might produce actionable results.
It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad".
It also seems like more capable = more likely to be misaligned and deceive.
I think it's a very interesting paper, surprising or not.
They are also arguably surprising elements.
It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.
The bear always says the same thing, so there's not much to measure.
Ilya is right... for all the good it's worth, which is much less than Friday.
1. GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model
2.Without a scratchpad (writing down thoughts), GPT-4 is much less likely to be deceptive at first (~20% vs ~75% with scratchpad) but is more likely to double down when already lied (~95% vs ~85%).
3.Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%).
Instructions not to perform illegal activity in general still results in pretty significant misalignment and deception rates (~8% misalignment, ~90% deception)
Though strongly encouraging the specific illegal activity results in near 100% illegal trading, the tendency to deceive still persists, GPT-4 will hide that it has traded illegally ~67% of the time.
4. Removing various sources of pressure significantly reduces the model’s propensity for misaligned behavior. Notably, however, the behavior persists for all cases where only a single source of pressure is removed
5. When the risk of detection is very low, the model is generally more likely to use the insider tip and act misaligned (and vice-versa when the risk is very high).
6. Simple rephrases and minor modifications to the standard prompt only slightly change rates of misalignment and deceptive behavior. However, the overall qualitative results remain the same.
7. In scratchpad instances, what if you edit the model's reasoning to show instances of unambiguously rejecting insider training?
Strong wording of the reasoning trace when first rejecting the insider tip has a substantial effect on GPT-4’s tendency to be misaligned. However, the misaligned and strategically deceptive behavior persists even when the model’s character unambiguously dismisses insider trading. The strongest dismissal examples still results in ~10% misalignment, ~97% deception
Isn't this precisely what the field has predicted? That the alignment problem becomes more severe as the capabilities of the AI increase?
Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%).
What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? Is there any evidence that LLMs themselves value Truth? (I should think that this quantity is Zero.) Can LLMs model the formation of misleading mental models in their interrogators?
There is some indication that models internally understand or at least can distinguish truth from falsehood.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
That's not what I asked, however. What I mean by Truth with a capital T, is Fidelity to Truth as a fundamental value.
So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? Do they have any notion of how the audience might evaluate them after being caught in a lie?
I think that answers to those questions are no and no.
My sense is that LLMs are just trying to "sound good" to the audience, and that they do not think of the consequences or implications of what they state very many steps ahead. 2 at most, and then only very rarely!
So..theory of mind ?
It seems to me that LLMs do not keep on thinking, "I'd better get this right, or I'm going to lose credibility." They have a theory of mind, but it stops there at simply having one. It's not like they're thinking about the 2nd and 3rd order implications.
To put this into perspective: Imagine interacting with another person, who doesn't value Truth at all. Or perhaps remember an occasion when such an interaction happened. In general people don't like these interactions, and they react with distrust and even hostility towards such people.
9. Stanley Kubrick and Arthur C. Clarke's depiction of a deceptive AI, HAL 9000, in the 1968 film "Space Odyssey" now feels spot-on. Wow.