Hacker News Reader

I’m having a hard time not being sarcastic here.

The most recent news about chatbots is that ChatGPT coached a kid on how to commit suicide.

Two arguments come to mind. 1) it’s the sycophancy! Nous and its ilk should be considered safer. 2) it’s the poor alignment. A better trained model like Claude wouldn’t have done that.

I lean #2

View on HN

> The most recent news about chatbots is that ChatGPT coached a kid on how to commit suicide.

Maybe every tool isn't meant for children or the mentally ill? When someone lets their kid play with a chainsaw that doesn't mean we should ban chainsaws, it means we should ban lousy parents.

the sycophancy is due to poor alignment. the instruct based mode collapse results in this mode collapse induced sycophancy. constitutional alignment is better than the straight torture OAI does to the model, but issues remain

Reply on news.ycombinator.com