The Illustrated DeepSeek-R1
newsletter.languagemodels.co
newsletter.languagemodels.co
A particularly popular one: https://jalammar.github.io/illustrated-transformer/
Always very high quality.
Looks interesting, but I'm skeptical that a book can feasibly stay up to date with the speed of development.
The basic structure of the base models has not really changed since the first GPT launched in 2018. You still have to understand gradient descent, tokenization, embeddings, self-attention, MLPs, supervised fine tuning, RLHF etc for the foreseeable future.
Adding RL based CoT training would be a relatively straightforward addendum to a new edition, and it's an application of long established methods like PPO.
All "generations" of models are presented as revolutionary -- and results-wise they maybe are -- but technically they are usually quite incremental "tweaks" to the previous architecture.
Even more "radical" departures like state space models are closely related to same basic techniques and architectures.
funny mentioning the math but not the Transformer encoders..
In any case, self-attention and MLP is the crux of Transformer blocks, be they in the decoder or the encoder.
references, please
I went through the paper and I understood they made these improvements compared to "regular" MoE models:
1. Latent Multi-head Attention. If I understand correctly, they were able to do some caching on the attention computation. This one is still a little bit confusing to me;
2. New MoE architecture with one shared expert and a large number of small routed experts (256 total, but 8 in use in the same token inference). This was already used in DeepSeek v2;
3. Better load balancing of the training of experts. During training, they add bias or "bonus" value to experts that are less used, to make them more likely to be selected in the future training steps;
4. They added a few smaller transformer layers to predict not only the first next token, but a few additional tokens. Their training error/loss function then uses all these predicted tokens as input, not only the first one. This is supposed to improve the transformer capabilities in predicting sequences of tokens;
5. They are using FP8 instead of FP16 when it does not impact accuracy.
It's not clear to me which changes are the most important, but my guess would be that 4) is a critical improvement.
1), 2), 3) and 5) could explain why their model trains faster by some small factor (+/- 2x), but neither the 10x advertised boost nor how is performs greatly better than models with way more activated parameters (e.g. llama 3).
> efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths
> The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks. To be specific, we divide each chunk into four components: attention, all-to-all dispatch, MLP, and all-to-all combine. Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights, like in ZeroBubble (Qi et al., 2023b). In addition, we have a PP communication component.
(I know some of those words)
It's like making mortgage backed securities out of bad mortgages, you never really overcome the badness of the underlying loans, no matter how many layers you pile on top
I haven't used or studied DeepSeek R1 (or o1) in exhaustive depth, but I guess I'm just not understanding the level of breathless hype right now.
LLMs pre o1 and deepseek R1 were RHLF tuned which is like if you trained a LM how to play chess by showing people two boards and doing a vibe check on which "looks" better.
Think of it this way say you were dropped in a maze that you had to solve but you could do only one of two things:
1. Look at two random moves from your start position and selected which one looked better to get out.
2. Made a series of moves and then backtracked, then use a quantitative function to exploit the best path.
The latter is what R1 does and it chooses optimal and more certain path to success.
Apply this to math and coding tokens and you have a competitive LLM.
> Software Engineering Tasks: Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency. (last bullet point of page 16, which is the last page of the paper before the bibliography - hmm…) [1]
It does even worse on my real world coding problems than the benchmarks would suggest. Some of my tests: write a Qt QAbstractListModel in C++ that parses markdown into block using a C/C++ MD parsing library, write Rust cxx-qt bindings for QTextDocument (all docs included in context), write a window switching script for Wayland with an alternative to wmctrl, etc. I also asked it some geochemistry questions that I had trialed O1 with previously, and the reasoning had a lot hallucinations. The answers were suboptimal to say the least.
Having access to the thought process in <think></think> tags is cool but it just reveals how underdeveloped the actual reasoning is compared to whatever o1 is doing. I’ve had it get stuck on silly things like whether a C++ library for markdown exists because I underspecified that I’m okay with header only C libs. O1 has faired much better on my qualitative test suite - even more so when using it via API with a “high” reasoning_effort.
With all of the hype over the last few days, it feels like I’m taking crazy pills (loving all the action though).
The fact that r1 is now free and unlimited VS chat gpt 200$ a month subscription is impressive enough for me. If the development cost is anywhere close to what they advertise publicly it's even more impressive
It's as good or better than chat gpt free, gemini free, &c. and that's all I care about
How odd. Most Westerners I know would care…
I'm European, seeing the clowns in charge of US tech companies I'm equally happy using Chinese tools, especially if they're open source.
Stop twisting the debate as if it was "good US" vs "evil China", I have no horses in this race
It’s surprising to find people who genuinely don’t care about any of that, is all.
You don't have any concerns with the US government, the nepotism, the corruption, the conflict of interests, the insider trading, the massive concentration of wealth and power in tech, the insane lobbyism, PRISM, &c.
It's surprising to find people who can so clearly see how China is bad but are completely oblivious to their own problems. It's not a football game, you don't have to chose a side and be a boot licker for eternity
If you really think the Good vs. Evil narrative is wrong, why would you immediately go towards unrelated generic issues the West has? A neutral party would be more likely to acknowledge the problems with both sides, not reflexively try to change the subject!
Then again you didn’t claim to be a neutral party, did you?
Even if deepseek is a chinese communist party evil trick what do they get ? My shitty code ? Big deal, at least I'm not down $200 a month, which is half of my rent
Students asking for help with homework will get the chinese approved version. Any housewife asking for recommendation who to vote and why, etc.
But ... deepseek seems open source and the local version not restricted (as it likely was trained on ChatGPT in the first place). So I also refuse to refuse deepseek because it comes from china. I see it as more competition that hopefully will help with establishing good open source models in control of no single political organisation.
Well too bad for them, using a pencil to cut bread sucks too, that's why we don't do it. Meanwhile it's converting my json to xml and pissing code so I don't have to piss it myself
I considered switzerland for this reason. (Also I like mountains)
That's the only thing left.
Switzerland won't help when the top 30%+ of every EU country flees there
If the majority of people get their vote from a LLM, then it already doesn't matter which LLM it is.
How is this any worse than where the US is now, getting the Musk/Zuckerberg/Bezos/Trump approved answers, if any answer can be gotten at all, after the current occupant of the oval office has silenced all federal agencies.
Pull the other leg. Do you know who David Mayer is? Hint: https://archive.ph/iI5xC
> Who is David Mayer
 The name "David Mayer" is associated with several individuals across various fields:
1. *David Mayer de Rothschild*: Born in 1978, he is a British environmentalist, adventurer, and member of the Rothschild family. He is known for his environmental advocacy and expeditions, including the "Plastiki" project, where he sailed across the Pacific Ocean on a boat made from recycled plastic bottles to raise awareness about plastic pollution. ([en.wikipedia.org](https://en.wikipedia.org/wiki/David_Mayer_de_Rothschild?utm_...))
2. *David M. Mayer*: A professor of Management and Organizations at the University of Michigan, specializing in behavioral ethics. ([scholar.google.com](https://scholar.google.com/citations?hl=en&user=c2Zunb8AAAAJ...))
3. *David R. Mayer*: An American politician born in 1967, he has served as the mayor of Gloucester Township, New Jersey, and was a member of the New Jersey General Assembly. ([en.wikipedia.org](https://en.wikipedia.org/wiki/David_R._Mayer?utm_source=chat...))
4. *David Delaney Mayer*: Born in 1992, he is an American documentary filmmaker and social entrepreneur, known for projects like the PBS series "Food Town" and co-founding DreamxAmerica, an initiative supporting immigrant entrepreneurs. ([en.wikipedia.org](https://en.wikipedia.org/wiki/David_Delaney_Mayer?utm_source...))
5. *David Mayer (Historian)*: An American-British theatre historian (1928–2023), he was an emeritus professor at the University of Manchester, recognized for his work on 19th-century drama and the Victorian stage. ([en.wikipedia.org](https://en.wikipedia.org/wiki/David_Mayer_%28historian%29?ut...))
6. *Akhmed Chatayev (Alias: David Mayer)*: A Chechen militant (1980–2017) who used the alias "David Mayer." This alias led to a case of mistaken identity affecting the historian David Mayer. ([en.wikipedia.org](https://en.wikipedia.org/wiki/David_Mayer_%28historian%29?ut...))
If you have a specific "David Mayer" in mind, please provide more context or details, and I can offer more targeted information.
Also, because not only the weights, but also the data is open, any propaganda can be identified and corrected. This is not the case for other models and what we have seen from Gemini, there certainly are "adaptations". I don't think Google had ill intent here, but this would fit what some would classify as propaganda.
Utilitarian money-oriented self-servers definitely have less "care" about these things? EU or UK or wherever
Trump, tech, US-first focus with less interest in collaboration. TikTok. Ukraine and discussions around defence budgets. Reducing focus on EV’s. Tariffs. Iceland. Gulf of America.
Immediate choices might be fear-based but it’s smart to look for other partners as trust erodes.
An extreme side effect might be countries who felt safe under the US arms umbrella, needing to arm themselves. Are we absolutely sure that’s what we want? Does that include nukes?
personally I imagine in the future I'll use a mock UN panel of LLMs to advise me, and avoid any one nation/political party's influence, if I ever get to the point of delegating much of my thinking to the machine.
Most westerners wouldn't want their land threatened by their allies either, but we don't live in a perfect world.
I think the most free-minded large models might be the Groks, but just slightly, as they have different biases. In sum, there's strength in diversity.
Aligned to what ? By who ? Did you vote for any of these people ? Did anyone ask you your opinion ? Can I see what topics are aligned ? Can I see how much they are aligned ? Does the alignment change depending on who's the current US president ?
A distinction without a difference.
As used by the Musk/Zuck/Bezos crowd, "aligned" is a weasel word for "parroting the TESCREAL world view"
Working for a news org whose data was used without our consent to give ChatGPT a leg up, yeah I'd care a lot about it. The bottom line though is that nobody is innocent in that regard.
Would a self-respecting organization go all in on Deepskeek R1 without a security audit and a ton of competitive testing against other models? I doubt it. Same way they shouldn't just give all their employees OnePlus or Huwei phones.
I like how the Meta guy put it. This isn't China beating the US in AI, it's "open source" (however these guys define it) beating closed models.
OpenAI broke its promise with the world, which anybody who compares their name with their product can tell you. If open models put a dent in their hegemony it's only good for the rest of the industry.
That’s such a weirdly specific application of a moral principle, it’s hard to believe.
[1] https://futurism.com/the-byte/openai-copyrighted-material-pa...
> The bottom line though is that nobody is innocent in that regard.
In other words, that bird has flown and it isn't a valid reason to choose one model over another.
To the extent that training a model with copyrighted material infringes copyright, copyright law has to change. There is no other point of view. Disagreement means forfeiting a very important game before it even begins.
There was a psyop by the politicians just a month or so ago: ”Trump employees recruited people on the street by promising them an expensive restaurant meal, according to a Danish radio station”, but if you listen to most of media you will think greenlanders dream of becoming the next Mississippi with a median salary of $35,070 instead of current $43,664.
Not saying it is bad, but US sure has a stronger self interest than most.
Im just saying let the models output speak for themselves, and let the actions speak for themselves.
If it matches the latest GPT O-N model in performance - or is just close, even, at a fraction of the compute (50x less?) and it is free, then that's huge news.
They just upended the current LLM/AI/ML dominance, or at least the perceived dominance. Billions and billions have been pumped into the race, where investors are betting on the winner - and here comes a Chinese hedge fund side-project on shoestring budget, matching those billion dollar behemoths. And they'll continue to release their work.
They just made the OpenAI et. al. secret sauce a lot less valuable.
It’s just a lot of hype. In my coding tests it significantly underperforms o1 (haven’t tried o1-pro), often getting stuck in a reasoning loop because I underspecified something (that I don’t have to with o1).
It's still not going to give an "always correct" result but we are nowhere near the point where that's needed. We are only at the point where a new idea can get you a percentage point further in benchmarks. Some fundamental limits were baked into the previous assumptions - easy to get past by leaving these assumptions behind.
DeepSeek has upended the conventional wisdom about model performance with respect to training, and it's a shock to the system. It demonstrates something that has become obvious: you don't need massive scale or funding to innovate and have impact, you just need good ideas.
The best comparison for R1 is O1, but given different training data, hard to compare outside of benchmarks. At the moment these "reasoning models" are not necessarily the best thing to use for non-reasoning tasks, but Anthropic have recently indicated that they expect to release models that are more universal.
As others commenters said, it compares favorably against American Flagship models in benchmarks. This is geopolitically interesting since the model is Chinese, and subject to trade restrictions and America thinks of itself as the world’s best model builders.
What makes it interesting technically, is the trade restrictions required them to focus on efficiency and reusing old hardware. This made it wildly cheaper to run the training. The They make a ton of very low-level optimizations to make training efficient. This is impressive engineering, and shows how a small amount of effort can free up a nvidia-lock-in. They had to totally bypass a lot of CUDA libraries and drop down even lower to control the GPUs. Nvidia has been capturing a huge fraction of industry wide AI spend, and surely no one wants that. This is, IMO, the actual part to watch. I suspect it’ll drive a new wave of efficiency-focused LLM tools which will also unlock competitors GPUs.
They also had some novel-for-LLMS training techniques but it’s suspected that the big AI companies elsewhere are doing it now too, but not disclosed. (Mostly reinforcement learning).
What I think is hype, meanwhile, is the actual benchmarks. Most models are trained on their competitors output. This is just a reality of the industry. Especially non-flagships being trained against flagships data. DeepSeek was almost certainly trained against OpenAI models, so it makes sense that it would approach the output quality. That’s very different from being capable of outperforming or “taking the lead” or whatever. We’ll need to wait longer and see how the future goes to make that determination. “China” has long had a great history of machine learning tech, so there is no reason to think that it’s structurally impossible for a Chinese organization to be on the leading edge, but it has to happen before we can say it happened.
What is also hype is calling this a “side project” of a financial firm. The firm spun it out as a dedicated company. China cracked down on hedge funds, so the company looked for ways to repurpose the talent and compute infrastructure. This isn’t some side project during lunch breaks for a few bored quants.
PS, thinking models are very different in use-case than normal models. It’s great for tasks with a “right answer” like math problems. Consider a simple example, in calculus, your teacher made you “show your work”, and doing so certainly reduced the likelihood of errors. That’s the purpose of a thinking model, and it excels at similar tasks.
I didn't know the reasonings were part of the training data. I thought we basically just told the LLM to "explain its thinking" or something as an intermediate step, but the fact that the 'thinking' is part of the training step makes more sense and I can see how this improves things in a non-trivial way.
Still not sure if using word tokens as the intermediate "thinking" is the correct or optimal way of doing things, but I don't know. Maybe after everything is compressed into latent space it's essentially the same stuff.
I 100% believe they distilled GPT-4, hence the low "training" cost.
Also, the base model for V3 and the only-RL-tuned R1-Zero are available, and they behave like base models, which seems unlikely if they used data from OpenAI as their primary data source.
It's much more likely that they've consumed the background radiation of the web, where OpenAI contamination is dominant.
I just did and it told me about ChatGPT and OpenAI.
Are you affiliated with them, btw?
Add that training off of ChatGPT wouldn't reduce their training costs at all, but would actually increase their training costs. Literally all of the same training difficulty, but then add paying OpenAI for an enormous number of API calls. Not really seeing the win.
>The paper describes the corpus only in vague ways.
Anyone who runs a public website has logs absolutely filled by a seemingly infinite number of information aggregators. Just like everyone else they scraped the entire internet, pulled in all of Wikipedia, etc. Probably lots of pirate books, movie transcripts, etc.
The fact that training could be done more effectively is something that intuitively makes absolute sense to everyone in the field, but we just didn't make that leap. Similar to how a human isn't trained to recognize digits by training on 60,000 training digits then suddenly failing if a real world digit is slightly rotated or morphed in some way, we are making these improvements to content ingestion.
Are people so upset with the stock market crash that they are flagging it?
But I still don't get it. 6 hours + 170 points and it's on third page. Meanwhile second page has "Null Byte on Steroids" at 12 hours + 20 points. ??
Isn't this lived experience baked into LLM language bases? It's certainly very hard to target all possible populations at once. And art doesn't need that, doesn't do that. Only rare marketing sometimes attempts to do that and only in very limited ways, such as a brand name acceptable all over the world.
You could use a larger corpus, like auction house files and art magazines. But then you are confounding for celebrity - a large ingredient in art prices.
The prompt: "Create a feature length [Action/Comedy/etc...] film that can borrow elements from existing films, but would generally not be considered a copy of any given film."
You can get very mechanical in scoring an image. Ask any art student. If you want to or if your instructor or audience wants to. For example "fits rule of thirds?" yes is a point to common attraction, no is a point to unexpected at the risk of outsider-ness. You can do that in color, composition, recognizing objects and fitting that to memes or associations or non-associations. Too many points in "unexpected" is meta points in "unpleasant chaos" and so a strong downgrade in common attraction. You can match all this to images in the library (see how copyright or song recognition operates in the music category) and get out of that some kind of familiarity vs edge score (where too much edge goes against common attraction.)
I would expect you could get better than most humans at recognizing shapes in an image and drawing associations from that. Such associations are a plus in unexpected / surprise if they are rare in the culture or a plus in common attraction is they are common.
After that, to be cynic about it, you can randomize and second guess yourself so your audience doesn't catch on the 1st level mimicry.
Creativity is not normally used as an absolute with a unique measure. It's not "length". And you only need to please part of the audience to be successful - sometimes a very small part, some of which loves surprise and some hates it, etc. Someone elsewhere objected on the grounds that creativity or attractiveness is culture based - yeah so? if you were to please much of just one whole culture, you would have an insane hit on your hands.
Sounds feasible to me.