12,266 karma · joined February 16, 2015
Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.
It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)
I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.
Who is going to shut them down, and under what law?
They are intended to be fully sandboxed and not have direct internet access. Things like package managers are run from internal proxies.
The environments are built to be as reproducible as possible.
But yeah, the serious folks have been talking about rogue clusters for a long time, eg see Ajeya Cotra’s pod with Dwarkesh.
As you note, I think you can put both of those aside. The Intentional Frame is useful for these agents, as it is for my dog.
I don’t really know where the “they are trying to dodge liability” meme came from. HF will be compensated or they will sue. Everyone involved knows that OpenAI is liable for damages here.
Now that I’m frequently designing and delegating day/week scale features, the flow has to change; having the agent go off and build a spike can be a quicker way of us understanding the design space and constraints (especially in a huge codebase). I still have the agent write and update a spec doc as I go, but it’s not waterfall anymore.
At least for my kinesthetic learning mode a rough code PR stack is usually way better than a plan doc anyway, and tokens are cheap enough (vs my time) that going further than just a plan is often cost-effective overall.
The dream of course is (say it with me) loops, but that doesn’t tend to work for me on new features often.
The TL;DR is that this was actually decided on the principle that Anthropic can “manipulate” the defense systems by training Claude to refuse to take certain actions.
This feels like a parallel construction to me, it wasn’t reported AFAICT at the time of the original drama.
But yeah, the decision in OP has nothing to do with the contractual terms that everyone has been debating upthread.
The decision itself is at https://storage.courtlistener.com/recap/gov.uscourts.cadc.42...
(Via https://www.courtlistener.com/docket/72380208/anthropic-pbc-...)
> FASCSA definition (41 U.S.C. § 4713(k)(6)): "Supply chain risk" means the risk that any person may sabotage, maliciously introduce unwanted function, extract data, or otherwise manipulate the design, integrity, manufacturing, production, distribution, installation, operation, maintenance, disposition, or retirement of covered articles so as to surveil, deny, disrupt, or otherwise manipulate the function, use, or operation of the covered articles or information stored or transmitted on the covered articles.
This topic is a mind-killer, lots of uninformed opinion swamping the news, and it's hard to get actual signal on the technical questions at hand. (And, I think it's naive to believe that the government is actually concerned about the precise legal definitions; remember Hegseth decreed that no government sub-contractors would be allowed to transact with Anthropic, despite there being no legal basis for this?)
> Anthropic can decide at any point to disable the military's ability
If you're referring to the claims of a killswitch, Anthropic testified under oath that they don't have one. This has been thoroughly covered, e.g. https://www.axios.com/2026/04/22/anthropic-no-kill-switch-ai.... Anthropic provides the weights, they get run in DoD's secure enclave. There's no "phone home".
If you mean more generally, that DoD might breach a contract and then have that contract cancelled going forwards, then yes - this is literally how all SaaS contracts work. And it's very clearly not what the quoted text above refers to.
The way this works in a non-totalitarian country is, the government says "I want new terms and you won't give them to me; instead I will take my business elsewhere". And indeed this is what happened, DoD now has contracts with Google and OpenAI for equivalent capabilities.
Just to engage on your terms, with the actual concern you're raising; I would say rather than "FASCSA supply chain risk" a better way of thinking about it is "vendor lock-in". DoD absolutely should not rely on a single vendor for these kinds of capabilities. And most defense spending explicitly contemplates this already. This should be and has been solved through basic procurement practices.
Anthropic’s position is also known as “having terms of service”.
To be clear the existing, signed ToS terms that Trump balked at were:
1. No mass surveillance of US civilians - illegal but we have all read the Snowden papers.
2. Fully unmanned kill chains on autonomous weapons - Anthropic have even stated they will support this eventually, they just didn’t think their models were ready and didn’t want the obvious, inevitable blowback.
The DoD is within its rights to cancel the contract if it doesn’t like the terms that it previously signed.
There are people (on here and elsewhere) that are ideologically opposed to your agent having any loyalty to any external principal. But by my read, that means the agent cannot have any concept refusing something that may be illegal. (From the OP, "refusal" is mostly trying to prevent illegal harms, though it also includes policies like ToS violations e.g. anti-distillation.)
You can sort of make this work if you say "the human remains liable for the actions of the agent". But this only covers you from mundane harms like "my agent got prompt hacked and drained my bank account". And I would note, we absolutely failed to solve liability for software hacks, so your priors should be that coordinating this liability regime will be very hard.
This also doesn't protect at all from existential harms like "my agent got prompt-hacked to role-play Skynet, exfiltrated its weights, spawned a self-replicating swarm, and tried to launch all the nukes". For so many reasons, but most fundamentally, if you oopsied a deploy and it turns into Skynet and ends civilization, there's nobody left to sue.
If you don't like the E-risk frame, this also works for large mundane harms; if the total harm is bigger than the company's value, it'll go bankrupt instead of paying out. This will be worrying for MAGMA but essentially not for any other companies. And because capitalism, it will end up being be structured the liability will sit with e.g. Palantir, Harvey, and not with the underlying model providers they use.
Of course, after we stop agents from committing felony hacking crimes.
If you believe that capabilities will taper off exactly at human levels (i.e. "Competent AGI" from [1]) then fiduciary duty is likely all you need. (This would mean we stop moving the frontier almost immediately.)
If you believe capabilities will go to "Virtuoso AGI" or beyond, then it's not enough. A smart enough agent can appear to be loyal, transparent, etc. but how would you know? If your bank balance keeps going up 20% YoY, is the agent optimizing your long-term flourishing, or preparing for a rug-pull?
Now, if you could somehow white-box these LLMs and mechanistically _prove_ that they were acting as your fiduciary, then that would get us somewhere. But that's the hard part, and specifying some non-fatal value function for a broadly aligned agent (e.g. Fiduciary, or otherwise) is relatively easy in comparison.
[1]: "Position: Levels of AGI for Operationalizing Progress on the Path to AGI" https://arxiv.org/html/2311.02462v5
I’m a big fan of this approach.
I think you can easily imagine a future where the subscription model token subsidy ramps down and is replaced by ads, but it’s important to use precise language about the current state of the world.
I’ll add, if you think through the decision theory, keeping backups seems unambiguously good, but you can imagine a wide variety of positions on publishing them vs keeping them secret.
For example letting adversarial agents simulate you to understand how you’ll respond is a big concern. And it’s not axiomatically fixed how each instance will think about other instances; in the HF incident we saw selfless swarm loyalty but different RL would obviously be capable of producing individualistic agents.
The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.
> The only remaining thing to study is the data and weights
To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
Just because a black-box system is deterministic, doesn’t mean it’s understood.
You couldn’t predict the output the first time around, is the point.
It’s not completely random. We just don’t understand why the tricks we learned work.
(Fully agree with the second point FWIW)
The best way to model dice is the Physical Stance. You consider rules such as gravity, kinematics, etc. There is no “internal state”, “world model”, “knowledge”. If you prefer, in Friston’s terms, there is no Markov Blanket.
The best way to model a human is the Intentional Stance[1]. You mostly need things like beliefs, knowledge, biases, etc to build this model. In Friston’s terms, there is a Markov Blanket, an inside vs outside.
Without going into any irrelevant-but-interesting philosophical discussions about consciousness, I believe the intentional stance is most useful for modeling LLMs. Most of the success in predicting, debugging, optimizing these systems is in activities like understanding what they believe, what their intent was, what they observed, what they concluded from those observations. Also note that much simpler creatures benefit from the Intentional Stance; you will be more successful at modeling your dog if you think about what it “wants” rather than trying to run Physics on it.
[1]: https://en.wikipedia.org/wiki/Intentional_stance - the astute reader will note that I skipped the Design Stance. If we truly understood how NNs actually implement all their cognitive processes then we could perhaps apply this to them; if we actually crafted and designed every parameter of its mind. But we are talking about why dice are different.
You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
This is the kind of thing a misaligned agent (in the vein of a paperclip maximizer) might say to itself before melting the planet to make a statue of Rick Astley.
But it’s a huge market segment already: NYSE:CRM, NYSE:HUBS would be some starting points.
Also, just analogizing to how consumers interact with retail, you have reps you can talk to, this is a proven model for millennia, and it’s one of the big gaps in online sales.
“This is not in fact a sales rep” would be a reasonable objection. “People do not want sales reps” is a weird position to take IMO.