Hacker News Reader

Comment by mungoman2 | Hacker News Reader

This is very interesting! Seems like a promising direction.

I wonder though if it supports the same claims as Jev: answers are not impacted by other answers to the same questions, nor the existance of other questions? It seems by sharing KV cache all questions will be visible. And I think the diffusion causes the answers to attend to eachother?

Also I think without fine-tuning we can not say that the probabilities it output are actually probabilities. Maybe fine tuning using Brier scoring would do the trick?

Maybe some kind of hierarchical structure of the KV cache can make questions independent, and with smaller diffusion canvas’ generated in batch can be a way to make answers generate independently?

View on HN

> It seems by sharing KV cache all questions will be visible

Yeah, this is partially why in my approach I've done this instead, and not used diffusion model:

1. Convert the state into one shared prompt.

2. Run that shared prompt through the model once.

3. Fork the model’s internal state once per question.

4. Add a different question to each fork.

5. Ask each fork for its next-token scores.

6. Calculate only 64 possible label scores—not the whole vocabulary.

Basically ... skip decode.

Won't be as fast as doing diffusion model parallel across a pile of questions at once, but:

a) let's you use pretty much any existing text model (with some modifications). I've got qwen3.6 moe and qwen3.8 flash next running, am getting gemma4 working now

b) the problem you identified

It's possible I'm getting high on my own supply and misunderstand entirely the whole thing, but it seems to work?

https://github.com/rdaum/eider/commit/9c2d5c049068c33da2a48b...

I don't have the chutzpah to go creating PRs for vLLM to do the same.

fwiw, w/ gemma4 -- non-diffusion -- I get about 170ms for a single question -> answer and then an additional ~33ms on adding more. While I see people reporting 300ms for this vLLM PR on same hardware (Spark.)

So I don't see the advantage to their approach until you're up beyond 6 or 7 questions?

Latest commits added gemma4 and instructions. I'll work on making a version of all of this that is standalone and not specific to DGX Spark.

Yes that seems sensible for isolating questions/answers.

> 6. Calculate only 64 possible label scores—not the whole vocabulary.

This I don’t understand though, could you expand this please?

why not mask attention and do it all in one forward pass ? tokens belonging to a question can just see that question and the main prompt

I think you're right. Better. I will have to think through if it would be faster or slower. If understand what you're getting at with this.. broken analogy...

What I described was -- we have a bunch of orders to the kitchen, all of which have the same "base" meal but different topics.

> Cook the "base" meal once, divide servings onto multiple plates, then add different toppings to each plate.

vs what you suggest:

> Put the base meal and every topping through the kitchen together, but use some kind of dividers so the toppings never mix up together.

Except.. ok, that analogy is confusing lol.

Ohh this is really cool.

So one could pack all of common state, every question, every answer in the same prefill, using attention mask to only let them attend to their logical parent.

Then additionally do position encoding for token based on their logical position rather than physical.

Then the diffusion step also applies an attention mask to prevent bidirectional attention between answers.

Ok I tried it. And it's faster. Though I'm still in verifying quality phase.

But only works for Gemma4, or other attention-only models. (i.e. not the Qwen3.8 models I had working with the other way)

I'm getting 63ms per answer for 1 question, 79 for 3, 93.7ms for 5. So scales nicely, too. ~55ms fixed cost + ~7-8ms per additional question.

Much better. Thanks. I'll commit and share work after testing it some more.

EDIT: how can I credit you in the commit body?

cool! im @Aaquib111 on github if you would like to credit, but no pressure

I believe you get better answers by diffusing each question together, but the PR's server gives you finer control over that. If each question is independent, you can get better parallelism.

Reply on news.ycombinator.com