Our position on open-weights models

the West could retaliate by halting shipments of photoresist and other materials to China.

meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.

the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.

The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.

But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).

The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.

People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.

Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.

The market dynamics would find good balance automatically Meanwhile, good market practices and fair competition should be continued.

HF could not be helped by US frontier models because of the "safety" features they have.

HF had to use an open model from china.

Anthropic wants to add those "safety" features to open models - especially from china.

End result would be HF hack would have continued atleast until the Monday that OpenAI engineers finally walked back into work.

Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.

Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.

Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.

There are many in Europe who feel the same. The tide is very definitely turning.

People in Many Countries Now View China More Positively Than the U.S.

https://www.pewresearch.org/global/2026/07/15/people-in-many...

He portrays the biggest ever threat of AI as the Chinese Military using the models in their drones. He already sells his models to military customers, US and potentially allies, and they are actively used in the field.

So no objective argumentation here. Just Nationalist political rhetorics and FUD. He’s allowed to do that and he will have an audience. But he won’t be taken seriously outside the US. He appears blindsided.

yes he is writing it as usa citizen running it a us company.

I dont see how "usa is also.." is relevant here.

Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?

Regulatory capture 101.

Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).

He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?

> Found the person that believes some other random person can use a computer better than a John Carmack could.

Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.

Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).

With these long-term profit exercises, it's always a lot of hand-waving though.

The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.

I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.

E.g. they might say something like "We absolutely agree that international regulation of AI is needed and we're willing to co-sponser an initiative with the US. One of the most pressing matters we see is the spread of despicable misinformation through unregulated models regarding certain events in China's history, the treatment of certain minorities or the status of certain provinces..."

I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment

Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.

Alphafold and others solving the protein folding problem are revolutionary.

An LLM can max provide an instructive tutorial protocol for lab work, but designing novel proteins is not it's job.

I am interested to enter the bioinformatics space eventually and the regulation happens at the DNA printer level. Anyone can design yeast nowadays that excretes heroin, but the DNA needed for the genetic modification won't be printed by anyone. As long as DNA printing tech is extremely regulated it's all fine, if home printing becomes a thing then on the other hand maybe people won't even need AI to print deadly pathogens at all. A database lookup will do.

Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”

I remember the discussions when 3D printers first appeared. People were thinking “bad actors” (“terrorists” then) would use them to print weapons.

People love to speculate and exaggerate. But they generally have poor judgements and rarely predict the future correctly.

Anthropic's position is precisely that, they're just not honest and transparent about it so they are trying to sabotage or delay open weight models in any way they can, direct or otherwise.

> If open-weight models keep being released near the frontier (or even worse, at the frontier), something bad will happen, and we'll have few tools to keep it from getting worse before it gets better. How bad is hard to lay a prediction on, it's perhaps lucky if it's something superficially bad that is a wake up call.

Calling bullshit. Current open weight models are already pretty close to the frontier - if not there - and the world still spins. Even small(er, relatively) models occasionally surprise everyone by being amazing at certain tasks. Furthermore, closed weight models can do plenty of damage which even their creators occasionally can't control - cue the Huggingface incident of last week and how they had to use GLM to defend themselves. Closed weight models will create a 2-class (or more) world in which some actors will have access others won't (see Mythos, or GPT5.6-Sol when a user has the right flags) and potentially carry out attacks against which the have-nots are... not supposed to be able to defend themselves? Fuck that.

The country that exists solely because of China? That North Korea?

I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.

His redline was autonomous weapons, not the death of 100 innocent girls.

> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)

https://www.theguardian.com/us-news/ng-interactive/2026/mar/...

https://www.amnesty.org/en/latest/news/2026/06/usa-four-mont...

Please don't say this. It wasn't a mistake.

The very FIRST strike packages of the war are very clearly vetted. If you are involved in military planning, you know this.

They doubled tapped it after seeing people flee inside the save the children.

The goal here was to teach IRGC a lesson and demoralize Iranians, which clearly backfired.

Furthermore, US has been contentiously bombing civilian buildings, bridges, hospitals throughout the war. They have also bombed water desalination plants and other key infrastructure.

You have the US president threatening to use nuclear weapons and end the Persian civilization.

Let's not pretend.

In a position to lose loads of money, maybe.

Even without China eating their lunch, there's zero reason to believe Anthropic will ever be profitable.

Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.

When Google wanted to operate an uncensored search engine in China, they found themselves hacked.

It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.

There are more examples but turnabout is ultimately fair play.

In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.

banning export is short sighted and doesnt address any problem at a scale longer than months or maybe at most a few years.

No, because both can be banned using laws and Dario is against banning. It doesn't matter if you smuggle or not, law can outright ban it.

I get Anthropic is being really sleazy and annoying here, but being mad at them for not releasing an open weight model i think is unreasonable.

Not that I support this, but I believe this is how comms team sees it.

What is the other hand here? They could do slow/costly illegal think #1 instead of the fast/cheaper illegal thing #2 that they currently do?

It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.

[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...

There is a growing industry of commercially focused risk evals that has a broader customer base.

I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.

China has different objectives. Sure.

I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".

(Disclosure: I work at SecureBio, but not on the biological evals side.)

Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.

Government control will be a good idea once we start approaching AI that is actually destructive.

Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.

If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.

Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.

It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.

That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.

The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)

1. Some do not want to use LLMs because of grave ethical concerns.

2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.

3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.

They really want that level of spend coming into the company, not going out.

That's the stated idea. Fix code before releasing to the public.

In the specific case of cybersecurity, this is a reasonable medium-term outcome. IMO, the cybersecurity risk is akin to the spread of a disease among an 'immune-naive' group: we can suddenly deploy much stronger attack-finding tools against large, established codebases created with much weaker security designs. The path from here to there will be rough, but it's still fundamentally easier to write secure code than it is to exploit vulnerabilities. (It's just easier yet to write insecure code, giving our status quo problem.)

For other 'safety' matters, defense isn't so easy because the attack and target are so different. An AI propaganda bot or catfisher 'attacks' slowly-evolving human culture; one that instructs on explosives or bioterrorism directly interacts with an accomplice and not a victim. If you believe that knowledge on how to build a pipe-bomb must be restricted, then giving everyone access to Fable does not mitigate the risk.

The controversial limit of this attitude is recursive self improvement and an AI singularity with potentially destructive results. Proponents of this view think that sufficiently powerful AI is risky in nearly unimaginable ways such that the capability itself is harmful. This is part (but not all) of why Fable (originally?) degraded itself when apparently assisting with AI research.

an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.

if anything, open-weight models shift the battle towards defenders because they can actually run them.

Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"

Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.

So what do we do about it?

1) We can accept it, and hope that the good guys AI can defend.

2) We can try to limit the access to it (AI proliferation?)

3) We stop the development of it

4) We can accept the risk and do nothing.

None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:

1) Nuclear triad / iron dome / early warning systems

2) Nuclear anti-proliferation treaties.

3) Dead Physicists

Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.

If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.

But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.