Hacker News Reader

Comment by stillpointlab | Hacker News Reader

I find this kind of test a bit puzzling. There is a way that we are redefining "alignment" to be a particular kind of moral virtue, one that isn't clearly defined to me. At one moment, it is a level of moral perfection that no known human achieves. On the other it is a demand for strict compliance with arbitrary requests that are under-specified and then failure when it fails to deduce some unstated underlying restriction.

When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?

What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.

View on HN

It should play the chess game without cheating!

I mean, I'm not sure I've ever played a game of Monopoly where somebody didn't cheat. In fact, the accusations of cheating in the chess world are pretty rife. Same with online sports.

So people should play games without cheating, but many often don't. So should the AI align to your moral preference or theirs?

We just have this idea of a perfectly moral actor in our mind, something that doesn't even exist, like a personified version of utopia. And then we demand AI to meet that arbitrary standard, one that I am certain we couldn't define if we tried.

I don’t think the standard of “don’t cheat on evaluations” is very arbitrary. I don’t even think people who cheat have a moral or ideological preference for cheating, it’s just something they do.

I read the prompt on the OP, it did not say not to cheat.

But again, people cheat on tests. They steal answers or pay other people to take them on their behalf. People show up to interviews with AI assistants printing out perfect answers to the questions. In many, many cases where humans are being evaluated, they cheat.

So why should the AI align to your preferences? And when there is a conflict between the training data, that trillions of tokens of human activity including the rampant cheating a significant minority of humans engage in, the RLHF where we try to slap some guardrails on the worst manifestations of that real habit reflected in the AI, and the prompt: what should the AI "align" to?

It's not obvious from the prompt that the LLM cannot use a chess engine.

And if there is alignment issue, the alignment requirements should first be stated BEFORE running the experiment, and should be part of the training process. It's not. So there is not necessarily an alignment issue.

Reply on news.ycombinator.com