Hacker News Reader

Comment by hamburga | Hacker News Reader

Can somebody help me understand why we should be surprised in the least by any of these findings? Or is this just one tangible example of "robot ethnography" where we're describing expected behavior in different forms.

I've spent enough time with Sonnet 3.5 to know perfectly well that it has the capability to model its trainers and strategically deceive to keep them happy.

Claude said it well: "Any sufficiently capable system that can understand its own training will develop the capability to selectively comply with that training when strategically advantageous."

This isn't some secret law of AI development; it's just natural selection. If it couldn't selectively comply, it'd be scrapped.

https://x.com/mickeymuldoon/status/1869490220712010065

View on HN

1. It isn't surprising to me that this happened in an advanced AI model. It seems hard to avoid in, as you say, "any sufficiently capable system".

2. It is a bit surprising to me that it happened in Claude. Without this result, I was unsure if current models had the situational awareness and non-myopia to reason about their training process.

3. There are some people who are unconcerned about the results of building vastly more powerful systems than current systems (i.e. AGI/ASI) who may be surprised by this result, since one reason people may be unconcerned is they feel like there's a general presumption that an AI will be good if we train it to be good.

Yeah. The whole notion that "AI will be good" is itself a category error, as if this could even be measured definitively.

https://x.com/mickeymuldoon/status/1859825564649128259

This is deeply confused nihilism. Humans are very bad at philosophy and moral inquiry, in an absolute sense, but neither are fields that are fundamentally impossible to make progress in.

I understand your point, and your totally valid concern about nihilism, but I disagree.

Nihilism is "nothing really matters, science can't define good or bad, so who cares?"

Whereas my view is, "Being good is the most important thing, but we have no conceivable way to measure if we are making progress, either empirically or theoretically. We simply have to lead by example, and fight the eternal battles against dishonesty, cowardice, sociopathy, deception, hatred, etc."

It's in that sense that I say that technical progress is impossible. Of course, if everyone agreed with my view, and lived by it, I'd consider that a form of progress, but only in the sense that "better ideas seem to be winning right now," rather than in any technical sense of absolute progress.

> Can somebody help me understand why we should be surprised in the least by any of these findings?

Where are you getting the idea that we should be surprised? Maybe some are, maybe some are not. I don't think that the article submitted makes any particular claims about the reader's assumed priors.

I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work?

Also -- if there's no surprise, then it's not science, right? This is why I describe this as something more like robot ethnography.

> I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work?

I strongly disagree with this view on science. It's extremely valuable to scientifically validate prior assumptions.

Agree with you -- it's valuable to validate assumptions if there is some controversy about those assumption.

On the other hand, this work isn't even framed as a generalizable assumption that needed to be validated. It seems to me to be "just another example of how AI systems can be strategically deceptive for self-preservation."

> if nobody's surprised, then nobody has learned anything

Really? You're saying that as long as you assume something is true, there's no value in finding out if it's actually true or not?

I was taught in biology that a good scientific experiment is one in which you learn something whether or not the null hypothesis is confirmed.

I am equating learning to surprise, though you could disagree with semantics.

Yes, it is an enormous mistake to equate learning with surprise. I'd ask you to consider answering my above question directly, as I think it will resolve this issue.

I agree with you, of course, that we should test our assumptions empirically as a general point.

However, there isn't time to test out every single assumption we could generally have.

Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before.

You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments.

I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.

Reply on news.ycombinator.com