Hacker News Reader

Comment by XCSme | Hacker News Reader

> You will notice multiple "thinking" parts per "turn"

I thought that was the code harness simply minifying the outputs. Many models now no longer return the entire chain-of-thought (to avoid distillation attacks). So yes, we don't get the raw LLM output, but I think it's just the thinking summarized, not a complex orchestration or different models.

I do agree though that now cloud models are kind of a black box, that's not only obfuscated but also changes over time. Companies seem to be changing model capabilities without notifying users, or even hiddenly serving completely different models. This is even worse via OpenRouter, with providers serving open-source models, some of them serve heavily quantized versions or even completely different models.

View on HN

idk what is "minifying outputs" in the context of what we are talking about. Opencode is opensource, you can find out what it is doing.

Last time I checked, OpenAI even send (in the response) the summary of the thinking part alreafy in markdown, so opencode has to remove the formatting to format it to their liking.

> Many models now no longer return the entire chain-of-thought (to avoid distillation attacks).

This is what they say: to avoid distillation attacks. And to some large extent this is true. I am saying there is a side- effect and this side- effect (depending on how tin-foilly you want to go) may be either a nice thing to have or it may be the "main reason" for all of this.

The side effect is splicing the inference, brokering requests, and what not, which brings huge benefits at scale.

This was my original point: openweights model to a sota model may be apples to oranges. So when will a local model catchup with its single cot run which is not even shaped properly: well never.

It is apples to oranges.

So, are you saying that local models are maybe better than we give them credit? Because with some extra orchestration/processing we could improve the results?

Yes, local models have already all that is needed, they have all the prerequisites.

But what they do not have is the correct shape, the correct approach. This is missing and it shows on multiple scales: it shows in the COT, it shows in the output itself, it shows in the infra to serve the models, it shows in the model orchestration.

This is what anthropic said one year ago:

> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. Users requiring raw chains of thought for advanced prompt engineering can contact sales about our new Developer Mode to retain full access.

Reply on news.ycombinator.com