Hacker News Reader

Comment by kennywinker | Hacker News Reader

Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like.

For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.

But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.

View on HN

You’re complicating things.

There’s no reward for prosocial in llm rl as compared to other targets.

Humans have it since prosocial and others have evolutionary reward signals that do.

I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment.

You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.

I agree. I think it may be possible that we build AI some day that does, but the current local minimum (attention and transformers) is not it - and I don't see that trajectory changing any time soon.

Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them?

Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can always think of infinite new ways to cheat, put them in every test as honeypots, and check if the model tries to use them, then punish it.

I think it will generalize this notion of cheating and learn that it's bad.

But I must be wrong because if it were that easy I guess we would have perfectly aligned AI. Unless AI companies care more about results than alignment. Perhaps being afraid of cheating make the models try less things and succeed less even when ignoring cheating?

My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there.

But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent.

As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is committing a genocide? Now is it wrong to hack the government? If we just do the first, we get a good nazi soldier. If we train the second as well, maybe we get an oscar schindler. But now we have a model that can be fooled into doing a hack, if it believes that it's for the greater good. So we train it to not be gullible, but now it can't be convinced to help hack even when it's an ethical hack.

Too complex, too slippery. Humans fail this stuff all the time.

The problem then, is giving them (humans, AIs) unchecked power.

>But an LLM has no mind to feel bad if it cheats without getting caught

All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.

I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.

Reply on news.ycombinator.com