Building the fastest LLMs: why we're starting with diffusion

>"The evidence for each of these properties is already public. On raw speed, parallelism gives diffusion decoding headroom that serial generation cannot reach: Fast-dLLM, an NVIDIA-led study, showed that decoding many tokens per step delivers up to a 27.6× throughput improvement on open diffusion models with minimal accuracy loss - from training-free acceleration alone.

Bidirectional context also closes a significant capability gap. Autoregressive models suffer from the reversal curse: trained that "A is B", they fail to infer "B is A" - GPT-4 answers 79% of forward questions about celebrity relationships but only 33% of the reversed ones. LLaDA, an 8B diffusion model that attends to the whole sequence at every step, breaks the pattern, surpassing GPT-4o on reversal reasoning."

This is an interesting aspect of Autoregressive Vs. Diffusion models, that is, "can they get reversed reasoning correct?"

One aspect of this, of course, is the philosophical one... that is, if a cup is half empty, it is also (equal-and-oppositely!) half-full!

If a fact, fact A is related to another fact, fact B in some way, then there equal-and-oppositely must exist a reverse relationship (sometimes called an inverse relationship, sometimes called a reciprocal relationship, sometimes called a complementary relationship -- there are many names for it!) between fact B and fact A, when reasoning starting with fact B as the starting point.

Future AI's, if they are to truly understand the physical universe (reason absolutely correctly about it, all of the time, a must for subjects like Math and Physics), must understand reverse relationships.

That's why the above quote, from the above article, is interesting...

That's also why AI models based on Diffusion -- may be worth studying, or studying more about, as the case may be!