On DeepSeek and export controls
darioamodei.com
darioamodei.com
1 - https://stratechery.com/2025/deepseek-faq/
2 - https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture
Anyway, this will always be a deep pocket race. But I wish it wasn't so much about brute-forcing GPUs and wasting power to (as yet) uncertain outcomes as far as model capabilities are concerned, and to me what DeepSeek achieved was to point out ingenuity and better techniques should be something that both OpenAI and Anthropic ought to be pursuing instead of burning cash.
Everyone seems to rag on OpenAI/Anthropic for spending so much money and take it as a symptom of capitalist waste, but this reality seems great to me - massive amounts of money is basically being funneled from VCs toward progress in machine learning. Once expensive breakthroughs are made, it is only a matter of a few years until people make the engineering optimizations to make those breakthroughs cheap.
Just want to emphasize the progress in cost that Dario highlights: fixed AI capabilities are becoming 4x cheaper every year. That is absolutely insane. US GDP growth averaged 3-4% over the last 250 years and look how far that has taken us. Moore's Law averaged ~40% annual growth in transistor density and look how far that has taken us in just 65 years. 4x growth is AI capability/cost per year is absolutely insane.
The "bad" news is that so far I've yet to see a serious contender emerging from that part of the world (although the Middle East has invested heavily in LLMs due to obvious cultural differences).
"To be clear, the goal here is not to deny China or any other authoritarian country the immense benefits in science, medicine, quality of life, etc. that come from very powerful AI systems. Everyone should be able to benefit from AI. The goal is to prevent them from gaining military dominance."
Navigating a world in transition: Dario Amodei in conversation with Zanny Minton Beddoes https://www.youtube.com/watch?v=uvMolVW_2v0
You can tell as an academic he wants to give DeepSeek props, at least that is what I would like to believe. The first third has the presenter leaning into "cheap chinese" too many times.
Dario makes an astounding implicit assumption:
- China originating labs cannot acquire chips providing 80-90% similar utility without the US within the next 2-3 years.
I'll make an observation, re: DeepSeek's incentives that drove them to create the innovations from the V2 and V3 papers.
DeepSeek, compared to American AI labs, are much more compute constrained, but in a unique way. Their chips are more memory bandwidth constrained (depending on type anywhere from 50% to 80% less bandwidth).
Therefore, each dollar/hour of investment towards memory optimization is worth MORE to DeepSeek than to American labs.
In the V2/3 paper, they've demonstrated exactly that with these memory optimization techniques.
1. MLA -> reduces KV cache by nearly 80% compared to GQA. By the way, this was published in V2 in May 2024.
2. FP8 matmul (while still accumlating in FP32 gradients) without losing significant quality.
3. DualPipe scheduling and reworking of Hopper SM's allocation on communication vs. computation -> DeepSeek's V3 paper has 2 full pages of hardware suggestions for "hardware designers" (read NVIDIA)
Export controls in a global market create different incentives in parties. The resulting incentives will change, and agents (using it as an traditional economics term) will change their capital allocation strategy.
Says the CEO whose product [1] costs 15-50x times more. (This is not just the DeepSeek's API, but also 3p providers hosting the same model)
> DeepSeek does not "do for $6M5 what cost US AI companies billions". I can only speak for Anthropic, but Claude 3.5 Sonnet is a mid-sized model that cost a few $10M's to train
Ok, that's still at least 3-10x cost reduction (assuming "a few $10M" lowerbound of $20M). And for a model that he later implies is 2x larger than Sonnet. So that's 6-10x efficiency improvement. Nice!
> Since DeepSeek-V3 is worse than those US frontier models — let’s say by ~2x on the scaling curve.
What curve? Does he mean the simplistic performance / model params curve? That does not take into account that DeepSeek v3 is a MoE (can't compare MoE and dense param # in a naive way), nor the other architecture changes (KV compression, etc.).
Also, if Sonnet 3.5 is 2x smaller, then why is inference 15-50x more expensive than DeepSeek v3's? Does Anthropic not have good GPU engineers? Are they just running at insanely high margins? As a consumer I don't care how big your model is behind the scenes. I care about API costs or inference efficiency when hosting the model myself.
[1] Product that is mostly comparable and in some ways quite ahead.
> Since DeepSeek-V3 is worse than those US frontier models — let’s say by ~2x on the scaling curve, which I think is quite generous to DeepSeek-V3
He says that it's 2x worse. So if it has the ~same quality [1], it would imply it's 2x larger. Unless I misunderstood what he meant there, of course.
[1]: "DeepSeek produced a model close to the performance of US models" -- in his own words.
This is just like Captain America giving a pre-battle speech to the Avengers—it's so inspiring! Hail Hydra!
What allies? Is he aware that his current President is alienating and antagonising almost every single country out there that can historically be considered an ally?
edit: (Please read this in a sarcastic voice, I think it's a crazy idea!)
that would be horrible, a multipolar world? what a nightmare
It may also have some positives, too, but situations with several similarly sized powers competing for primacy have historically been pretty bloody. Unlike Apple and Google, the US, China, Russia etc. have military forces at their disposal.
Multipolarity doesn't imply we go back to the 1910s. The idea would be to strengthen multilateral institutions that put a check on things like the World Wars.
If there is anything that keeps us from tearing each other's throats out, it is a) democracy, which in most countries makes going into a war of aggression somewhat harder (I know, the US is a huge exception, but Trump I. was partially elected on the basis of Clinton being perceived as a war hawk), and b) the intuition that weapons are now so destructive that there is nothing to win, except scorched earth.
Still that didn't stop Putin from launching his war; miscalculation such as his ("the enemy is a paper tiger and will fold immediately") is very possible even today.
IIRC the same delusion led Saddam to invade Iran in 1980.
no, it's just because some countrys lost their power of aggression, nothing related to democracy
the US is not the only one with dirty hands, every NATO member is part of it
just check France, the only reason it doesn't colonize african anymore is it's too weak
not a single french voter gives a fk about what france is doing in africa
i mean, all 'rule-based international order' died in history, the Holy Roman Empire, the First Empire of France, and the ancient Chinese empire died in every 300 years
even there is no China, even the empire doesn't implode, will africans and south americans be willing to mining for the west for all their lives? there will be Sankaras and Castros in every several decades, every hegemony has an end
or, american exceptionalism?
I mean, the fact that the Romans and the Chinese never clashed over future Kazakhstan is not a proof of a peaceful multipolar world. It is simply a corollary of the fact that neither empire was able to expand into the steppe.
and, i know you guys don't like it, but that's what communism aims for
it's an extremely glorious and great undertaking that has never been done by our predecessors
The same with chips, if no sanctions on exports to NVIDIA, we would have access to Huawei Ascend chips (used by DeepSeek to have lower runtime costs)
Wow!
There are popular infographics floating around today which explain to the layman that deepseek invented MoE.
https://arxiv.org/html/2401.06066v1
The red star in the first chart is so funny.
Or like they see it and have learned the appropriate weight to give unsupported predictions of this type from people with a vested interest in them being perceived as true. It not only not new information, its not information at all.
and, personally, i think, if any CEO in this industry dare to say "we won't get a super AI in 2028"
many people will be disappointed, some people will be scared, and one prisident be pissed off
Also, if the US seeks a permanent advantage over China by 2027 with self-replicating AI, so much so that it can dictate whatever terms militarily. Wouldn't that just force China to send a few cruise missiles to TSMC before that happen?
Maybe it's worth listening to a Western AI researcher who worked for a Chinese company and came to the conclusion "China must not beat us to AGI/ASI".
Why on earth would they do that? China considers Taiwan to be rightfully theirs and would consider TMSC to be a valuable asset.
Given Moore’s law and the efficiency that will clearly come from optimizing chips for AI and competition increasing the amount of VRAM on these devices to run models locally.
Is creating an export regime today going to mean that in 3 or 4 years general smart phones and high end laptops are all going to be subject to export controls?
Keep in mind that at one time computers that consumed entire building are less powerful than my apple watch.
The H20 with 96GB of memory is currently available in China. We're a ways off from restricting consumer devices.
There's a reason why Oppenheimer & co. weren't non-stop publishing internationally accessible op-eds during WWII about how important it is that Germany not develop a scaled uranium fission bomb before the US.
People _really_ ought to study more of our past.
The only way this would work is to create a kind of great firewall of America keeping open source Chinese models out while the remainder of the globe uses something like DeepSeek.
It would literally be the dumbest soft-power move in the history of the US.
Prediction: they will do this as soon as they see sufficient domestic chip production.
The attempts to inshore fabs to the USA is too little too late.
And it doesn’t matter if Trump defends Taiwan and makes China back down - the fabs will have been bombed to bits.
As it is, trump would probably just do sanctions and tariffs, which - like Russia - China will expect to weather.
What would prevent the US from directly attacking China's domestic fabs in this scenario? Defending Taiwan doesn't seem as proxy-ish as Ukraine considering China considers it their territory already.
Both mainland and Taiwan belong to China, it's written even in the Taiwanese constitution. The controversy is in the definition of China, basically they are supposed to unify mainland while the mainland wants to unify them. A balance will be reached at some point either peacefully or violently.
Peace is more likely if the US stops pouring oil on fire.
The US should stay out of this situation if its leadership has its own people's interest in mind. A hot war between US and China will cause many millions to die and possibly end of human civilization. Let that sink in.
Yes, I'm aware. My reasoning follows from my response to the sibling of your comment. I think the US is piuring gasoline on the fire to use ww3 as a mechanism to reset the national debt.
It seems even more unlikely that, should the US be prepared to bomb land targets in China, that they would do anything beyond overt military targets that are being used to attack Taiwan.
When trump has recently been asked what he’ll do when China attacks Taiwan he has just said sanctions and tariffs.
My going hypothesis is that the US cannot politically solve their Modern Monetary Theory national debt blackhole problem and is therefore itching for WW3 to be able to "reset" debts when the dust settles. Taiwan seems like a good opportunity for that, but maybe there is some sanity left in Congress.
Clearly china could last longer under similar conditions, but they're also looking weaker than they did 5 years ago.
1) we need to do various forms of regulation to entrench US closed source market leaders, which happens to increase his company’s value
2) the best way towards improvement in models is not efficiency but continuing to burn ever increasing piles of money, which happens to increase his company’s value
China spends 1.5% of its GDP on its military. The US spends 3.5%. I get that the two countries are engaged in competition for dominance, but why is China the bigger threat here?
Also, there are a great deal of groups and institutions out there that are pushing for more diplomacy, more cooperation, and a ratcheting down of tensions. If Dario is going to get political anyway, why not go that route?
https://investors.palantir.com/news-details/2024/Anthropic-a...
As an outsider looking in, I’m not thinking of the USA as a particularly democratic country any more. And I didn’t think its export controls were about democracy, but rather a trade weapon.
Necessity is the mother of invention. Deepseek is demonstrating that if the incentives are aligned to want efficiency then good engineers - and there are plenty of them in every country - can make things fly.
What an incredible claim to just slip in two-thirds of the way through! Where have I heard it before?
1965, H. A. Simon: "Machines will be capable, within twenty years, of doing any work a man can do."
Excellent analysis and I largely agree. The part where he looses me here is the certainty in the claim around fundamental changes to the economics...
This is open weights with a (largely) open recipe. Arguably, Meta has already been at it but now there are two labs producing frontier models at this level. A wishful analysis could imagine that there _might_ be more and that _could_ result in a qualitative shift in how the frontier model research is being done and hence affect the economics but Dario seems to have largely ignored this aspect.
Not sure why?
The simplified export control analysis Amodei gives here gets a lot more complicated when increasing export controls potentially ends up spurring additional R&D in Nvidia competitors.
How about everyone has access to the best model because it's Open Source and openly developed?
And then there goes the usual techno-feudal interjection:
>If China can't get millions of chips, we'll (at least temporarily) live in a unipolar world, where only the US and its allies have these models. It's unclear whether the unipolar world will last, but there's at least the possibility that, but there's at least the possibility that, because AI systems can eventually help make even smarter AI systems, a temporary lead could be parlayed into a durable advantage
So Dario envisions a unipolar world where a black box corporation (his own one, of course) controls a perpetually self-improving machine god and distributes scraps to the masses that are too unwashed and dangerous to control it themselves, while at the same time preventing other countries from breaking out and making their own self-improving machine god because they're evil. A boot stomping on your face forever, regardless of who you are.
That's coming from an "ethical, helpful, and honest" guy.
>To be clear, the goal here is not to deny China or any other authoritarian country the immense benefits in science, medicine, quality of life, etc. that come from very powerful AI systems. Everyone should be able to benefit from AI. The goal is to prevent them from gaining military dominance.
And of course he immediately tries to deny that thought.
I'm sure Anthropic's valuation got hit badly by the DeepSeek saga, and the CEO is lashing out in panic.
And it amazes me how many people can't seem to see past the "US is good, everyone else are bad" smoke mirror.
"Eschew flamebait. Avoid generic tangents."
Second- aside from the specifics of this post, you don't have to believe that the US is good to see the logic of export controls. There are a number of countries that are openly hostile to the US (Iran and North Korea), and also some that are semi-privately working against the US (like China). These countries will take any advantage that they can- they will steal IP and use the US's development resources to catch up to the current state of the art at a reduced cost. So you don't have to put this in terms of good vs. bad, just think of it in terms of things that benefit the US and things that don't benefit the US. Whether you think people are evil is irrelevant, it is very logical for the US to put export controls on things that it doesn't want to give away for free. The US wants to preserve its advantage and make it as costly and difficult for these other countries to catch up.
They also aren't imprisoning people for speech, which is important when we're discussing who we want controlling AI in the future.
The entity that is least problematic (of the available options) and should be entrusted is the European Union.
Don't forget Central America and Iran; supporting military dictators in Southeast Asia (Marcos in the Philippines and Suharto in Indonesia); invading Haiti and raiding it of the entirety of its gold reserves, condemning its people to poverty -- one could go on and on.
They absolutely did directly and indirectly several times in their history. A few of those times were recently:
> In September 2020, it was revealed that ICE had performed mass hysterectomies on immigrant women in several detention centers, reminiscent of the long-standing US policy of sterilization of black and brown women. 2
> The US currently operates a system of slave labor camps, including at least 54 prison farms involved in agricultural slave labor. Outside of agricultural slavery, Federal Prison Industries operates a multi-billion dollar industry with ~ 52 prison factories , where prisoners produce furniture, clothing, circuit boards, products for the military, computer aided design services, call center support for private companies. 1, 2, 3
> In the present day, ICE (U.S._Immigration_and_Customs_Enforcement), the police tasked with immigration enforcement, operates over 200 prison camps, housing over 31,000 undocumented people deemed "aliens", 20,000 of which have no criminal convictions, in the US system of immigration detention. The camps include forced labor (often with contracts from private companies), poor conditions, lack of rights (since the undocumented aren't considered citizens), and forced deportations, often splitting up families. Detainees are often held for a year without trial, with antiquated court procedures pushing back court dates for months, encouraging many to accept immediate deportation in the hopes of being able to return faster than the court can reach a decision, but forfeiting legal status, in a cruel system of coercion. 1, 2
> During the 2020 coravirus pandemic, it was found that a law that empowered police to arrest those for not social distancing, lead to 80% of those arrested being black and latino.
etc., etc., etc.
https://github.com/dessalines/essays/blob/main/us_atrocities...
When the controlling party of the US is discussing concentration camps for immigrants while also happily calling to revoke birthright citizenship and deport citizens of the US that criticize Trump or the Republican party (eh - who am I kidding - it's the maga/heritage party now)...
It feels like we have no legs left to stand on here, and the support was DAMN shaky to begin with, seeing as we were routinely knocking over governments that we see as inconvenient in our geopolitical sphere of control for the last 100+ years.
Essentially, I think your argument is about 3 elections stale.
https://github.com/dessalines/essays/blob/main/us_atrocities...
So it makes sense to him to argue for export controls using whatever rhetorical flexes and flourishes he can come up with.
This situation is temporary, of course. China previously had a large incentive to get their own leading-edge nodes into production, and now they have a Manhattan Project-size incentive.
But TSMC is building new leading edge fabs right now.
Like, any global hegemony will be increasingly corrupt given the power that gives, IMO.
Choice is - USA or China. And I certainly know in which world I'd rather live.
I will freely grant that Euro money goes to completely obsolete heavy industries and luxury items rather than technology innovation (and blame all politicians for missing out on technology), but there are outliers like ASML that almost make up for the whole thing...
Can't remember which countries where bombed by China in last 50 years, but I can give you plenty of examples for the USA
2A provides access to weapons. It doesn't allow people to murder one another.
> but where access to food, shelter, healthcare and education are all expensive privileges subject to the whims of the free market,...
Article 1 Section 8 doesn't enumerate Congress the power to provide any of those things.
What's more, I beieve the Federal government shouldn't. Rights should only ever be negative unless being a counterparty to positive action taken by the government (e.g., it makes sense to provide public defenders since it is reactionary to the government trying to take away your rights).
Healthcare, education, etc. are better served by the private sector with charity as the supplement for the less fortunate.
> and where you don't even need the popular vote to elect a President (it's just nice to have) because the electoral system is a series of 18th century compromises to keep slavers happy...
The electorial system is designed to prevent tyranny of the majority and is biased towards inaction. This seems better than a pure democracy to me because most government action is bad: inefficient, introduces moral hazard, etc.
In a system of pure democracy, the bottom 51% can vote to tax the top 49%'s income away. Then the next iteration the new bottom 51% can vote the same. Eventually this could lead to purely even distributions. That seems wildly worse for society: all incentive to produce would be void.
But most of what the government does introduces dead weight to economic transactions or moral hazard. This is especially true when the government tries to provide social benefits beyond taking over a product or service that could perfectly well be provided by the private sector.
I would have little argument against the government regulating that private healthcare providers cannot discriminate on the basis of preexisting conditions. It would increase costs uniformly across the industry. I do have arguments for providing healthcare on the basis of income, as I believe the program would be better served by private charity. Those who argue the contrary do so only because they want to force all citizens to contribute to the private charity (run publically) of healthcare-for-the-poor, because they want to solve someone else's problem but don't want to have to bear the burden themselves.
I hope one day, China starts blocking advanced tech from US, for the good of the whole world
Are we still capitalists?
I think the answer is - no.