Beyond Semantics: Unreasonable Effectiveness of Reasonless Intermediate Tokens
arxiv.org
arxiv.org
It’s a misconception that transformers reason in token space. Tokens don’t attend to other tokens. High dimensional latents attend to other high dimensional latents. The final layer of a decoder only transformer has full access to entire latent space of all previous latents, the same latents you can project into a distribution of next tokens.
That's essentially the core idea in Coconut[1][2], to keep the reasoning traces in a continuous space.
But yeah, the LLM doesn’t even know the sampler exists. I used the last layer as an example, but it’s likely that reasoning traces exist in the latent space of every layer not just the final one, with the most complex reasoning concentrated in the middle layers.
Reasoning is definitely not happening in the linear projection to logits if that’s what you mean.
There's a lot of stuff that makes this hard to discuss, ex. "projectable to semantic tokens" you mean "able to be written down"...right?
Something I do to make an idea really stretch its legs is reword it in Fat Tony, the Taleb character.
Setting that aside, why do we think this path finding can't be written down?
Is Claude/Gemini Plays Pokemon just an iterated A* search?
[1] https://en.m.wikipedia.org/wiki/The_Unreasonable_Effectivene...
But it would also be more unreasonable to dismiss the fact that a significant portion of the research community (and even greater portion of the public) was operating under these beliefs: that COT was akin to thinking (it's literally there in the name...). It is possible to disagree with something but also not believe someone is being unreasonable by coming to different conclusions.
It is rather "unreasonable" to think we can explore the world simply through pen and paper, from the comfort of a chair. You'd think you'd need to go out and touch grass, but incredibly this is not necessary.
| The first point is that the enormous usefulness of mathematics in the natural sciences is something bordering on the mysterious and that there is no rational explanation for it. Second, it is just this uncanny usefulness of mathematical concepts that raises the question of the uniqueness of our physical theories.
-------------------------------------
I think is fine. It is being used in a similar way to Winger, with similar context.
I can see two camps. One has always interpreted the COT as analogous to a model's internal dialogue. While the other has always thought there's a much larger gap between the manipulations within latent representations and what has been decoded, not necessarily needing be strongly aligned.[3] To the former, the results here would be shocking, while to the latter it is "yes, and?" Clearly they're addressing the former camp. There were plenty of people that Winger did not need to convince.
I'm of the latter camp[4], and I'm happy people are not just asserting and are demonstrating. Honestly, I'm even frequently upset when works get dismissed because they "demonstrate something we already knew" but no one had ever actually demonstrated. The proofs and evidencing is more important than the answer. Quite often we're highly certain about results but they are difficult to even evidence (let alone prove). I mean it would be quite silly to dismiss a proof that P != NP, even though the vast majority of us have long been convinced that this is the relationship we'll end up with. Yet, no one's done it.
-------------------------------------
[1] https://math.dartmouth.edu/~matc/MathDrama/reading/Hamming.h...
[2] https://static.googleusercontent.com/media/research.google.c...
[3] Both camps can be further broken down too. Lots of nuances and opinions here and the lines really get fuzzy as we try to make it more accurate. I don't want to pretend there's a hard defining line, but the distinction helps the discussion and I think is reasonably accurate enough. Let me know if you think it is a gross mischaracterization.
[4] I can expand more why this side seems "obvious" to me. But a warning, you can probably guess I'm not good at being terse.
[Note]: I'd even go so far as say we should revisit Winger's argument around AI. I'm certain mathematics can be and will be "unreasonably effective." But not enough time has been dedicated to formulate the right type of math to use. We really do have to invent a new kind here. This may sound weird to non-mathematicians, but even physics uses multiple kinds of mathematics. The operations, fields, and algebras you use in one part may not be appropriate in another part. That's okay. But we don't have a TOE yet either, and that's a critical part of finding a TOE, is bringing all this together.
I think you misinterpret what it's about. He's pointing out how remarkable it is that the universe obeys laws like E=MC^2 exactly as far as we can tell which is not something you would probably expect just from looking at the world. The pre scientific understanding of the world was it was driven my gods and spirits. The mathematical laws were only discovered by scientific investigation.
Or as he puts it:
>The miracle of the appropriateness of the language of mathematics for the formulation of the laws of physics is a wonderful gift which we neither understand nor deserve.
If he was just saying use maths it would be boring and not a famous paper 65 years on.
> I think you misinterpret what it's about. He's pointing out how remarkable it is that the universe obeys laws like...
> The pre scientific understanding of the world was it was driven my gods and spirits.
| "Well, now you are pushing your joke too far," said the classmate, "surely the population has nothing to do with the circumference of the circle."
I really did mean it when I said "about something that would sound silly today". We take this for granted now, with 60 years of working under this framework, but this wasn't always so. It seems silly now because much of the math we learn is in science classes and even outside we have a particular focus of teaching math most relating to science, but this is a small portion of a much larger field. Fwiw, I am not saying this as a complete outsider, I have a degree in physics.
It is also worth paying attention to the fact that Wigner helped create Mathematical Physics[0]. "Mathematical Physics" is not a pleonasm.
Don't take it just on my word! The Wiki page says something extremely similar!
| In it, Wigner observes that a theoretical physics's mathematical structure often points the way to further advances in that theory and to empirical predictions. Mathematical theories often have predictive power in describing nature. [1]
| Wigner argues that mathematical concepts have applicability far beyond the context in which they were originally developed[1]
> The mathematical laws were only discovered by scientific investigation.
Yes, he was saying "use maths". Yes, it sounds silly today, but so do a lot of things that needed be said in the past. I see no reason that the (now) obvious claim by Copernicus would make him any less famous.
[0] https://en.wikipedia.org/wiki/Mathematical_physics
[1] https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
You have "use maths" which sounds silly, I take it as "the appropriateness of maths is a miracle we don't understand" which is deeper and still largely true.
I really encourage you to read that wiki page.
| The quantum theory of the Lamb shift, as conceived by Bethe and established by Schwinger, is a purely mathematical theory and the only direct contribution of experiment was to show the existence of a measurable effect. The agreement with calculation is better than one part in a thousand."
| Hamming argues that Albert Einstein's pioneering work on special relativity was largely "scholastic" in its approach. He knew from the outset what the theory should look like (although he only knew this because of the Michelson–Morley experiment), and explored candidate theories with mathematical tools, not actual experiments. Hamming alleges that Einstein was so confident that his relativity theories were correct that the outcomes of observations designed to test them did not much interest him. If the observations were inconsistent with his theories, it would be the observations that were at fault.
It is kinda crazy that a thing we created, without the specific intent of modeling the world, ended up being so great at modeling the world. That's the unreasonable effectiveness.
In fairness, to change my opinion, you would need to show me some chain of reasoning or a conversation Wigner is clearly responding to that involves religion. Because this is what I see, but around math not being physics, and is what drives my interpretation.
[0] https://intellectualmathematics.com/opinionated-history-of-m...
Now, in comparison, nothing in this thread is going to annoy you!
But I think is a good example that fits the OP's critique (I don't think the critique fits to the arXiv paper. Even though I expected the main results, see my main comment).
The "unreasonableness" in Karpathy's post[1] is using sequencing to process non-sequential data. But the reason this isn't unreasonable is that we explicitly expect non-sequential processes to be able to be reformulated as sequential ones.
The SVHN (hose numbers) he shows is actually a great example of this. We humans don't process that all at once. Our eyes similarly dart around, even if very fast. Or we might think about how to draw a picture. We don't do everything at once, but we work in sections, building up, and have layers that end up being ordered even though this technically isn't a requirement. I'm actually struggling to think of things that cannot be broken down into sequences. He says as much here
| an important point to realize is that even if your inputs/outputs are fixed vectors, it is still possible to use this powerful formalism to process them in a sequential manner.
[0] https://static.googleusercontent.com/media/research.google.c...
[1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/
This is the interesting part. We've probably all had the experience where the model is going off the rails during the thinking process but somehow spits out the right answer at the end. Apparently the reasoning doesn't even need to be correct during training?
I guess it suggests to me that the reason CoT helps is that the model gets more compute to think internally, not that the words it produces are meaningful. I'm surprised nobody has come up with a good scheme for adaptive compute per token yet. Maybe we can skip CoT entirely.
How do we know if the reasoning was correct or not? Do we have more information about what the model was thinking besides just what it says it was thinking?
CoT builds on existing prompt engineering techniques by adding it to reinforcement learning to force the models to build their own CoT prompt essentially. So it's not what it's thinking but all indications are that it does guide the reasoning abilities of LLMs through the output distribution.
I have one, I just don't have the time or money to research it :(
- https://physics.allen-zhu.com/part-2-grade-school-math/part-...
The mock conversation that is written between think tags is not a conversation. It's the collection of tokens that are most likely to be written after a prompt to a model that was trained on example conversations.
Why is that different? In a real conversation, participants use logic to choose what is worth saying next. The next statement is already determined in the speaker's mind to be logically sound. In a mock conversation (the LLM's CoT), there is no logic. The next statement is only determined to be statistically familiar, then written immediately.
The end result of a desirable CoT interaction is text that would have been written by a thoughtful/logical conversationalist. Whether or not the mock conversation itself is logically consistent with the mock conclusion is irrelevant, because the LLM is only concerned with how familiar that mock conclusion is to the prompt, its mock conversation, and its training.
The overall vibe of how something is written behaves as a replacement for actual logic. Logical deduction is replaced with measures of confidence, conversations turns, etc. in writing style. It all works out in the end, because we are so consistent with the style in which we write real logical deductions, we have ended up providing an invisible semantics for the LLM to follow.
There is something meaningful that we are entirely blind to. Unfortunately, it doesn't follow rules the way logic does, so it's not a trustworthy replacement. Fortunately, it's useful for more general exploration.
It's no different than how in English we can signal that a statement is related to a kind of politics or that it's about sex through particular word and phrase choice.
Training for reasoning should be expected to amplify the subtext, since any random noise in the selection that by chance is correlated with the right results will get amplified.
Perhaps you could try to dampen this by training two distinct models for a while, then swap their reasoning for a while before going back-- but sadly distinct models may still end up with similar subtexts due to correlations in their training data. Maybe ones with very distinct tokenization would be less likely to do so.
I hope that research into understanding LLM qualia eventually allow us to understand e.g. what it's like to [be a bat](https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F)
We have our own personal 'culture' too-- it's just less obvious because its tied up with our own hidden state. If you go back and read old essays that you wrote you might notice some of it-- that ideas and feelings (maybe smells?) that are absolutely not explicitly in the text immediately come back to you, stuff that no one or maybe only a spouse or very close friend might think.
I think it may be very hard to explore hidden subtext because the signals may be almost arbitrarily weak and context dependent. The bare model may need only a little nudge to get to the right answer and the you have this big wall of "reasoning" where each token could carry very small amounts of subtext that cumulatively add up to a lot and push things in the right direction.
An example of this is toki pona, a minimalist constructed human language that is designed to only express "positive thinking". Yet it is extremely easy to insult people in toki pona: e.g. sina toki li pona pona pona pona. (you are speaking very very very very well).
To be free of a potential subtext sidechannel there can be essentially no equivalent outputs.
So the solution to this is to build a multi modal LLM that accepts both text and raw logits as input.
You query a model to produce the reasoning trace, then you feed the reasoning trace's logits into a second model query to produce the answer.
These steps have to be done separately, because there is no training data that contains the logits. You can only parallelize training when it is done on known input + output data.
Does you see your result as putting that paradigm in question, or does the explicit reasoning assessment perhaps ameliorate the issue?
https://arxiv.org/abs/2502.18600
Similar level of results in a fraction of the tokens resulting in similar quality for less cost for longer runs.
But also when interacting and needing to read the token responses I can read shorter responses way faster so my own speed is faster.