Hacker News Reader

I also believe this. Post-training LLMs with vague metrics can only be achieved with RLHF, which is not impossible, but extremely costly and difficult. Instead, companies will opt for RLVR, focusing on math and programming tasks. This pushes objectives away from writing quality; often far away. That is why older models, in my view, actually read better than newer ones. It's by design.

View on HN

You can brute force it by making it try random stuff then judge itself on it. You don't have to always use an LLM's output. Sometimes you can use that plus other things to add flavor. An LLM is actually really good at judging if something is good or bad. It just has a really hard time coming up with new things. But if you had unlimited compute you can throw in some rng and whimsy and get something resembling what humans do.

You can prove that doing this will spiral training into a fixed point. There was a lot of research into getting this to work in the past, but it never truly worked well. The hope was that if RLVR was used quite a bit, and the general performance crossed some threshold, that it would then be possible. However, since it has been shown that RLVR only concentrates the distribution of outputs rather than truly shift it, I doubt this will ever be a viable strategy.

Reply on news.ycombinator.com