Continuous Diffusion Language Models (CDLM's)
sander.ai
sander.ai
I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
(The only exception I will make is encoder-decoder models which now are often done by decoder-only.)
But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass.
Before ChatGPT there really wasn’t much of a concept of pre-training and post-training. It was all pre-training. Post training was what made the bots conversational and not just “continuing the thing you wrote to them”.
So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.
This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat.
Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies.
They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.
Government regulation was really not in the picture either in 2020 or 2021 tbh. The government was still trying to beat a pandemic. Some of the Biden admin eventually wanted to but it wasn’t very serious.
Imagine if CRISPR was considered "too dangerous" to publish because of the potential ethical ramifications, and that only a special few should be aware. It is utter self-righteousness, and it is shameful behaviour. The world cannot adjust itself to what it cannot see, so you risk greater catastrophe by keeping it secret.
The open dissemination of knowledge at every increment is the only way for society to truly deal with what is to come.
In case you were not aware, strong base models (no post training at all) have been available for quite some time now. Including ones that eclipse “scary” frontier models from even a year ago.
Words out of a magic box on a computer can’t make you kill yourself. Perhaps the people that would use that as encouragement are already mentally ill enough that it really doesn’t matter what the trigger is?
On a more callus but fully serious note, evolution starts out physical, that the species that don’t eat and breed as well, die. What happens when you remove that? When life is so safe that you basically wont stave, get eaten, catch a disease, don’t really need to compete all that hard for scarce resources, etc? Do you think evolution just stops? Or perhaps does a social or mental evolution become the predominant differentiator for successful reproduction over time?
My bet is 2027 we see an agentic worm make global headlines.
It's a different sort of danger. Knives, guns, etc are locally dangerous. The threat from AI is globally dangerous. Despite being a much lower risk of it happening, the blast radius is massively bigger, which means the overall danger is greater.
Also, maybe both things some be considered. Many countries manage the sale of knives, guns, etc. You don't need to accept that risk just because AI exists, or vice versa.
Which turned out to be true.
I'm tired of allowing stupid and gullible people to determine how the rest of us live, work, play, and create.
You are hyperbolically judging a post-hoc cherry-picked subset of a great spectrum of concern, with hindsight experience nobody had at the time.
Safely navigating the future isn't an accuracy contest. Risk mitigation has to account for the distribution of costs for different signs of error, where novelty, uncertainty, and any potential for compounding effects all greatly multiply the need for hedging.
He's not talking about decoders, he's talking about auto-regression. Before ChatGPT, the dominant paradigm was fine-tuning BERT-like models.
> Before ChatGPT there really wasn’t much of a concept of pre-training and post-training.
Again, people spend years just post-training BERTs in various ways.
> people spend years just post-training BERTs in various ways
Yes, I was one of them. That’s not called “post-training” it’s called fine-tuning.
I thought that fine-tuning was changing the weights in the model, not the embedding? Or did I misunderstand?
For example you might have one model which embeds a text query and another model which embeds an image. You also have a dataset of image + text captions. Training means updating the weights of those models so that the embedding of the image is close to the embedding of its corresponding caption.
For what it's worth, I don't believe post-training diffusion models (of language or otherwise) is uniquely difficult, just a lot less thoroughly explored so far.
And industry really cares about solving practical problems. So as soon as autoregression “solved” the text generation problem (albeit imperfectly) the field was ready to move on. If text diffusion is not faster, cheaper, or higher quality, there is no ROI.
In fact text diffusion is much faster, so I have used it in the past. And I think it has the potential to be better as well, given a constant time budget. So I think it can be a really promising method moving forward.
Right now models need to rely on less legible compressed CoT to get high intelligence per token/step, but with diffusion they would just need to output more tokens per step instead.
also haven't read too deep into the deepseek agent harness but the math in there was really cool. it sounded promising, at least.
Yes, same with openclaw etc. Some might have plugins to integrate with graph or vector databases besides only markdown files.