Hacker News Reader

Comment by bondarchuk | Hacker News Reader

Diffusion on images is easy to understand for me: you start with noise, the model denoises by shifting the pixels towards their final value. What is the equivalent operation for increasing or reducing noise in language here? Is the "noisy" sentence half-way through training or inference sort-of-correct but not really, and at 90% almost-correct but with slightly wrong words (semantically)? Is the noise somehow semantic at all or is it something else?

View on HN

For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.

How is that diffusion? What noise is it denoising there?

Don't think about defusion as denoising, but rather as learning a delta operator, or rather, the inverse of one. I don't know how diffusion language models work precisely, but if I were to haphazard a guess, I would say you may think of a sentence as a matrix of values, the operator as simply filling a line with zeros, and the inverse — what the model learns — as adding it back.

This is equivalent to cutting an image in blocks, and learning how to generate incrementally images by inpainting missing blocks. This in-painting mind you can be generated in multiple steps, where you incrementally add more into the block.

Example, as I understand:

(I'm not sure how should look prompt, my guess): Prompt: answer, what word is missing in text query. Query: What is it denoising there?

Not familiar at all with diffusion LLM but I'd guess you'd have noisy logits.

Reply on news.ycombinator.com