Hacker News Reader

Comment by dilyevsky | Hacker News Reader

> There really is no way to control them

it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough

View on HN

Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature".

It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").

The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter?

I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.

Now you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)

wouldn't that imply some sort of "subconscious" reasoning ability that's hidden? and if that was the case why would we need cot in the first place?

A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.

What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.

But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.

If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.

If we’re comparing it to human thought process then there’s famously s1 and s2 thinking with former being more like looping in the latent space and s2 being more like cot. Personally im not convinced the way llm “think” is all that analogous to real thinking and cot and looped transformers are just neat tricks to tease out pre training data better

Yes, although as far as COT goes there is no need to assume any similarity to human thinking. It's just about how many sequential steps - how long of a "chain of thought" - are needed to reason through and respond to any input.

Those steps could either be all internal (looped), therefore hidden, or with partial external visibility due to emitting a token every N steps. The complaint about Astra, and moving in the direction of hidden COT, is that it makes these models far harder to monitor and debug.

I agree that most of what these companies doing - synthetic data, etc - amounts to trying to squeeze all the juice out of the pre-training data, although the recent trend of RL post-training via agents running in custom task simulation environments does change that a bit - these environments, and the rewards they provide, are a new source of data.

how? There are no cot trajectories afaik. Each token is generated independently. "Analyzing the cot" is a natural language problem. One which we still don't know how to reliably solve. Besides throwing another defective model on it and calling it a breakthrough, of course.

there is a number of different methods. One is a separate model/classifier trained with human annotation. Here's a paper by OAI: https://arxiv.org/abs/2305.20050

A separate model/classifier is just using the same broken tool to fix the broken tool. A model is not an appropriate solution.

> A model is not an appropriate solution

that's a bold statement without any supporting argument. more like an opinion really. specially trained classifiers can be very good. sentiment analysis has been a thing for a while now, there had been entire businesses built on this tech where being even slightly wrong can be very costly

You have a car whose engine occasionally dies when left to idle. The fix for that is not to have a second car with the exact same problem give it a bit of a tow every once in a while.

"very good" is the exact same level advertised as the base models they're "protecting". It's not good enough, otherwise we wouldn't be having this conversation in the first place.

Reply on news.ycombinator.com