Hacker News Reader

Comment by geraneum | Hacker News Reader

Even with heavy RL post training and rubrics, the model is still fundamentally bound by the next token prediction mechanism at inference. Rlhf and cot just affect the probability distribution of which tokens get predicted next. Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.

View on HN

> and a single hallucinated token can still derail the entire chain of thought.

Incorrect. As the OP said, that is a very 2023 understanding of how LLMs work.

Grab a new model from OpenRouter. Have it work on a task. Change a few tokens and have it continue the completion.

> Have it work on a task.

With or without a harness?

Have you actually tried this yourself? Of course it can derail it. Try to reflect on your interactions with LLMs without all the constraints like web search, agentic scaffolding, etc.

The same way that a “yes” or a “no” input from you can change the response, cot tokens are fed back into the model as input and can derail it.

Have you? Can you show such a derailment with a large SOTA model?

It would be interesting.

I have seen such derailments within the GHCP harness maybe with GPT 5.6 Luna that went into some loop about whether it already provided a final response to the user, or 5.6 Sol suddenly switching to talking about MS SQL performance.

I also saw a post about Sonnet unexpectedly talking about Minecraft after seeing a file with a related name. The user thought it was the output of another user's conversation so the post was fairly popular.

> Harness… seeing a file…

Thank you for making my point for me. But let’s keep the goalposts stationary. We’re talking about LLMs without scaffolding.

Indeed, and it would be interesting whether it is much more likely to derail outside of a coding harness like in my examples.

I still don't know if that is the case, and how frequently it happens, since you did not share details beyond vaguely suggesting it would happen.

Without a harness. Yes, I just tried this on OpenRouter.

What did you try exactly?

When you speak or type, you speak one word at a time. When you move, you actuate one muscle at a time.

Does this mean that a single incorrect word or twitch will completely derail the task you’re trying to performance? Or will you, like any other intelligent being, recognize it and compensate?

Analogies are good for conveying meanings not proving statements. How human muscles or brain works has no bearing on LLMs.

> Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.

With reasoning models, a derailed chain of thought can be rerailed.

> can be rerailed

What rerails it?

The model can realize it made a mistake earlier and correct itself in subsequent output. I have seen it happen many times in CoT.

That’s the point. It may or may not. It can derail it.

This realization is something you assign meaning to. For the model there’s no difference between either of these states.

Reply on news.ycombinator.com