Hacker News Reader

Comment by vinkelhake | Hacker News Reader

I don't get where the author is coming from with the idea that a diffusion based LLM would hallucinate less.

> dLLMs can generate certain important portions first, validate it, and then continue the rest of the generation.

If you pause the animation in the linked tweet (not the one on the page), you can see that the intermediate versions are full of, well, baloney.

(and anyone who has messed around with diffusion based image generation knows the models are perfectly happy to hallucinate).

View on HN

Bidirectional seq2seq models are usually more accurate than unidirectional models.

However, autoregressive models that generate one token at a time are usually more accurate than parallel models that generate multiple tokens at a time.

In diffusion LLMs, both of these two effects interact. You can trade them off by determining how many tokens are generated at a time, and how many future tokens are used to predict the next set of tokens.

Hallucination is probably a feature of statistical prediction as a whole, not any particular architecture of neural network.

Regarding faulty intermediate versions, I think that’s the point. The diffusion process can correct wrong tokens with the global state implies it.

I think the discussion here is confusing the algorithm for the output. It's true that diffusion can rewrite tokens during generation, but it is doing so for consistency with the evolving output -- not "accuracy". I'm unaware of any research which shows that the final product, when iteration stops, is less likely to contain hallucinations than with autoregression.

With that said, I'm still excited about diffusion -- if it offers different cost points, and different interaction modes with generated text, it will be useful.

The Llada paper: https://ml-gsai.github.io/LLaDA-demo/ here implied strong bidirectional reasoning capabilities and improved performance on reversal tasks (where the model needs to reason backwards).

I made a logical leap from there.

I'm not sure about hallucination about facts, but it might be less prone to logically inconsistent statements of the form "the sky is red because[...] and that's why the sky is blue".

Reply on news.ycombinator.com