This article is trying to make the headline sound as if they replicated the model, when they “just” replicated the theory of one of their findings. It’s a great result even without the author’s obfuscation.
They're using a similar model and a similar training method to do something simple. It is to DeepSeek what DeepSeek is to ChatGPT-4o, and this kind of reduction to the simplest case is exactly what makes progress go.
(e.g. ChatGPT-4 is a dead end because it is so expensive to train that you can't do training experiments. At $30 a model it is reasonable to train a model 1000s of times)
But how they built on other open concepts like Cold Start + RL + Rejection Sampling etc... is the big thing.
I am not at all connected to whomever this is, but as I am on a "confidential project for a confidential client" it seems to simplify what I would overcomplicate anyway.
Some of us need to improve domain specificity, and are more interested in targeted capabilities.
DeepSeek-R1-Distill-Qwen-32B itself is pretty good to be honest, but it is more aspirational how easy it is to add reasoning to any base model IMHO.
Those of us who remember Simon’s satisficing principle, realize that often complex problems sub-optimal solutions.
We don't need, want or expect some central omniscient and omnipotent AGI, we want tools that are sufficient to solve real world problems.
Obviously having good reliable domain specific CoT for cold start etc... is hard.
But not nearly as hard as hitching the companies success to MS/OpenAI.
While there is a lot of noise...I think some people are missing the forrest for the trees here. Deepseek is more about moving the decision to use OpenAI for some needs from a single source vendor to one of convenience for us.
For those of us who are in the camp that OpenAI was never going to reach their "AGI" claims because of fundamental issues like the frame problem, this is huge.
Ya...we won't be training our own models for some silly budget of ~$5M like the press is claiming.
But we won't have to for some use cases...
I think it will take time and someone on a project with not such a strict NDA to demonstrate just how much this 'completes' things in ways that RAG, functions etc... didn't.
Or I just possibly got lucky with my needs? who knows.
It was orders of magnitude more before DeepSeek.
[0] https://newsletter.languagemodels.co/p/the-illustrated-deeps...
Well yes, yes you did, because all you had to do was follow the existing instructions, guidelines, and use easily available materials. You didn't go down any dead ends, didn't have to work your way from propellants like high test peroxide, dangerous hypergolics, and eventually develop solid rocket boosters.
It's like making the generic of a drug someone else developed.
Even if you merely flipped a single bit and created AGI based on existing tech, you're still the legitimate creator of AGI.
DeepSeek never claimed to have trained the base models they used. Now maybe a lot of people inferred that they did, but that's not what they claimed.
Their breakthrough was more along the lines of, "given a model, we can train it to reason using a certain class of RL techniques". Which is, to my mind, more useful in any case.
But yeah, if there are people out there thinking they can train base models from scratch for USD6 Million with no data, they're likely to be disagreeably surprised when they make the attempt.
Isn't that exactly what they have done? Maybe you are confused with the distilled models?
DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base"
DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"
They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat).
They trained their Zero model into a Chat+Reasoning model, which is what is attracting news.
They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
There are two papers: https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.
https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation.
I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.