After reading the article it seems to me that this is more like synaptic pruning where weak connections between neurons are eliminated in order to increase the efficiency of the neurons. Interesting to see that this also works for LLMs.
During the learning stage we want input from every variable so that we are sure that we don't omit a variable that turns out to be essential for the calculation. However in any calculation a human does 99.9999% of variables are irrelevant (e.g. what day of the week it is, am I sleepy, etc), so of course the brain wouldn't use resources to keep connections that aren't relevant to a given function. Imagine what a liability it would be if we have had excessive direct connections from our visual processing system to the piece of our brain that controls heartrate.
The unfortunate reality is that no one truly understands how memory works. Many theories are floating around, but the fundamental components remain elusive. One thing is certain: it is quite different from backpropagation. Thankfully, our brains do not suffer from catastrophic forgetting.
The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusion just focuses on 2B and 7B models like gemma and qwen.
An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm.
Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.
The authors did the equivalent of "Lets design a human intelligence benchmark, and use a bunch of 12 year olds as reference points"
I will eat my hat if the authors rescind the paper in a year or so if their benchmarks show no difference on SOTA models.
No definitive answer yet, but my bet is on no.
Large models successful now have dodged recurrent architecture, which is harder to train but allows for open ended inference steps, which would allow straightforward scaling to any number of reasoning steps.
At some point, recurrent connections are going to get re-incorporated into these models.
Maybe two stage training. First stage, learn to integrate as much information as well as possible, without recurrence. As is happening now. Second training stage, embed that model in a larger iterative model, and train for variable step reasoning.
Finally, successful iterative reasoning responses can be used as further examples for the non-iterative module.
This would be similar to how we reason in steps at first, in unfamiliar areas. But quickly learn to reason with faster direct responses, as we gain familiarity.
We continually fine tune our fast mode on our own more powerful slow mode successes.
Still 5k points to go, though! :D
Those models (4o, o1-mini, preview) don't see any drop at all on those benchmarks. The only benchmark that see drops with the SOTA models is the one they add, "seemingly relevant but ultimately irrelevant information".
Humans can and do drop in performance when presented with such alterations. Are they better than LLMs in that case ? Who knows ? Because these papers don't bother testing human baselines.
- after the answer, ask it "are you sure?" (from the office tv series: "is it a stupid thing to do? if it is, don't do it") - chain of thought, step-by-step thinking - different hats (godfather style: piecetime vs. wartime consigliere): looking at the problem from different points of view (at the same time or in stages). For example, first draft: stream of consciousness answer, second iteration: critic/editor/reviewer (produces comments), third (address comments), repeat for some time - collaborative work of different experts(MoE), delegate specific tasks to specialists - [deliberate] practice with immediate feedback