Hacker News Reader

We need to recognize this as a failure in training. It did some useful stuff but it can be much better. A training signal is likely missing.

View on HN

As I understand it, the rough guess as to what's happening here is that most recent capabilities progress comes from specific verifiable-rewards reinforcement training (RL). The RL pressures are all about task performance, but (surprise surprise) highly human-legible English language usage isn't very important to the models abilities to address the tasks.

Weirdly enough, the pressures are having them drift toward novel dialects of English that work well for their own chains of thought. Open question about whether they'd drift all the way to a new language given enough time.

This is tricky, because we really want language-independent training of skills. We know that self-play type of reinforcement learning is incredibly effective when possible. But at the same time, they are our tools - so we need supervised language training for this reason? It's possible that training just needs to be rebalanced so that RL with rewards is balanced with rounds of language adjustment. And to really make that happen, benchmarks need to score the models on that.

Reply on news.ycombinator.com