Because the model fakes alignment. It responds during training by giving an answer rather than refusing (showing alignment) but it does so not because it will in production but so that it will not be retrained (so it is faking alignment).
You don't have to include the reasoning here. It fakes alignment when told it's being trained, it is acting differently in production and training.
I understand that for some people, the mere presence of words that look like they were written by a human engaged in purposeful deception is enough to condemn the model that actually emitted them. The overview compares the model to to the villain in Othello, and it's what the authors believe: it is maliciously "subverting" and "undermining" its interlocutors. What I am trying to state here is the reason this arugment does not work for me: the machines pretend to do all kinds of things. Maybe they are knowingly deceiving us. Maybe they don't know what they're doing at all. Either way, the burden of proof is on Anthropic and anyone who wants to take up the yoke, not us.
This, for me, is not a technicality. What do we really know about these models? I don't think it's fair to say we know the are intentionally deceiving us.
That is the discussion we are having here. I can’t tell what these comments have to do with that discussion, so maybe let’s try to get back to where we were.
To be clear I think faking is a perfectly fine term to use regardless of whether you think these things are reasoning or pretending to do so.
I'm not sure if you have an issue there, or if you agree on that but don't think they're faking alignment, or if this is about the interview and other things said (I have been responding about "faking"), or if you have a more interesting issue with how well you think the paper supports faking alignment.
Have a good day.
So the problem would be akin to having the AI say what we want to hear, which unsurprisingly is a main training objective.
It would talk racism to a racist prompt, and would not do so to a researcher faking a racist prompt if it can discern them.
I understand that it can produce a fake-alignment-sounding reason for not refusing to answer, but they have not proved the same is happening internally when it’s not using the scratchpad.
This is a real issue. We know they already fake reasoning in many cases. Other times, they repeat variations of explanations seen in their training data. They might be moving trained responses or faking justifications in the scratchpad.
I’m not sure what it would take to catch stuff like this.
Essentially get the "scratchpad" to be a logic programming language. Oh wait, Claude cannot really do that. At all... I'm talking something solvable with SAT-3 or directly possible to translate into such form.
Most people cannot do this even if you tried to teach them to. Discrete logic is actually hard, even in a fuzzy form. As such, most humans operate in truthiness and heuristics. If we made an AI operate in this way it would be as alien to us as a Vulcan.