Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR.
This is because it can edit and doesn’t suffer from early token bias.
This is because it can edit and doesn’t suffer from early token bias.
> Gemini Diffusion’s external benchmark performance is comparable to much larger models, whilst also being faster.
That doesn't necessarily mean that they scale as well as autoregressive models.
With the diffusion transformer and newer architectures, it might be possible that transformers can now be applied to diffusion. Diffusion also has the benefit of being able to "think" with the amount of diffusion steps instead of having to output tokens and then reasoning about them.
I think it's hard to tell exactly where we are headed but it's an interesting research direction especially now that it's somewhat more validated by Google.
# d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
# Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning
Could you please clarify that?
AR is not the only possible solution, but many other techniques floating around do not have that property of actually learning the right thing. Moreover, since the proposed limitation (not being able to think a long time about your response before continuing) is a byproduct of current architectures rather than a fundamental flaw with AR, it's not as obvious as it might seem that you'd want to axe the technique.