RLHF Is Cr*P, It's a Paint Job on a Rusty Car: Geoffrey Hinton

Of course it has fundamental problems. Likely they are more prevalent then in other techniques, regarding LLM training or otherwise.

But one also needs to acknowledge how immensely useful and powerful the resulting models have become, achieving results pure supervised fine-tuning could likely not have achieved at all. Whether fundamental alignment of pre-trained models is even possible is a very different question.

Related: 'Fine-tuning on human preferences is a fool’s errand'

https://lifearchitect.ai/alignment/

> Hinton said in a recent interview. “You design a huge piece of software that has gazillions of bugs in it. And then you say what I’m going to do is I’m going to go through and try and block each and put a finger in each hole in the dyke,” he said.

While I agree with the underlying sentiment, that transcription, uh... I change that to "dike".

Anyway, it seems while we can round off a lot of rough edges from an LLM, it's fundamentally so much of a dream-machine that there's always something else, a never-ending game of whack-a-mole.

It gets even worse when it comes to malicious or adversarial stuff, and not just "ensure it doesn't say generate a rude word most of the time."