Dario, Please
pop.rdi.sh
pop.rdi.sh
I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity?
Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).
Agents could make a virus that does not require continued inference to do it's thing.
Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
Also, AI providers are literally getting a stream of traffic with every prompt and every response. How can they not know what's being worked on? They are more likely to use that an excuse to ban open models where they can't know what's being worked on.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.
As humans understand them, anyway. As long as we're hallucinating up magic computer viruses, RSI dictates that the AI agents are keenly aware of GPU RAM sizing, and will design a useful model to fit into what's readily available, with headroom for context and tool calling, far better than I could do as a human. But magic doesn't exist and AI still needs to follow the laws of physics, so maybe a model that can pass ExploitBench but do absolutely nothing else can be quantized down to fit on a 4080 GPU and still get a decent score on similar tasks, but there's a bitter lesson about that to be had.
I'm not sure I get what you mean by billing. These companies are running their own data centers (or are currently building them out). This could look as subtle as one machine giving slightly worse or slower answers.
Even though they're pretty bad at it, AI companies need to make money, or at least keep track of their expenses. Datacenters are expensive to operate, so they need to ensure that every instance of their models are either allocated to a paying customer session, or are being used for a legitimate purpose internally. If a session is running for a long time without justification, that's costing electricity, wear, and preventing allocation to better purposes. Billing is is the most reliable aspect of monitoring in the same way that the IRS is the most reliable part of the government.
This... just... doesn't matter. There are ways to scale horizontally at the expense of latency.. token/sec may drop dramatically, but then you just make millions of slow instances and in aggregate, you're back in action as a very powerful coordinated swarm...
You know cables, modems, RF equipment and optical transducers can all be unplugged right?
Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.
Because investors have pumped hundreds of billions into AI and real consequences put that money (and growth) at risk.
Another part is a completely defanged administration she it comes to effectively regulating anything.
Another bit is money.
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
Obviously this will have huge impact on some companies valuations, but you can have one's cake and eat it too.
That aside, I'm not sure why it's particularly interesting they have all this "compute" (let's just assume for the sake of argument it's all "live"--that is they can actually run workloads on all of the "compute" they have on paper). So what if it's the biggest amount ever? Why would that be meaningful? Is there some economically viable problem you're aware of that is somehow dominant in that way?
So I think part of what is going on now is not just the method (LLMs) but the means (supercomputer levels of compute) at unprecedented scale of concentration.
However, if I put my sci-fi hat on for a second, it's not so far fetched that we figure out a way to compress models to a size where it wouldn't need all that compute.
I, for one, am not trying to find scapegoats or go on a witchhunt.
But managers are paid a lot of money to take responsibility. Yes, that's an old school thought, responsibility. But that's one big reason they get a big, fat paycheck.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Imagine we made nuclear weapons a private industry, had CEOs bragging how they have enough warheads to blow the Earth to smithereens, and then they "accidentally" nuked three cities over a short period of time each, saying they lost control, or rather couldn't contain their semi-autonomous weapon. All somehow managing to turn the PR around from their abject incompetence and towards SciFi visions of mankind hunted by self-replicating bombs.
Countries demand reparation for damages in war. Citizens of those countries sue for damages and win.
Accountability is not a foreign concept. And the point is to disincentivize negligence. Because negligence is cheaper. And in this case, accidental hacks are marketing spend.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
Obviously you’re going to keep the model behind an API and be very selective about the people allowed to call and the queries it’s willing to answer, in that case. As Anthropic has done with Fable. But that is voluntary restraint - mostly in today’s regime we get frontier capabilities in open weight models on a ~year delay.
There is no user-involved damage. No one is recklessly running agents by the thousands without air-gapped containers, except “the people” than run these labs.
What defines a model? What defines ownership of a process? If I make a wrapper to a remote VM that builds and executed a prompt, am I accountable?
When I worked at a company in the EU, it was enough to apply a reversible linear transform to the data for it to be considered GDPR safe-according according to legal definition as long as the transform details were stored separately.
Hell, you can slip and fall and hold the cleaning company accountable. (This might be a US thing. Likely because that fall might cost a lot in medical expenses, and your insurance will do whatever it takes to pass the liability.)
"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.
You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?
The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1/2 layer that could increase transport costs at the Tier 2/3 layer that impacts ordinary people's bandwidth.
If I build a robot that murders my neighbor, I’m still at fault.
We don’t absolve drivers of responsibility because of cruise control.
In that sense, AI is nothing new. If it is abused to cause harm, the person behind it should be liable.
What if a person physically broke into the car and did the same thing? Clearly they are the one to blame then.
The whole person in the loop is liable is already an outdated concept when decisions are made beyond the persons physical control.
If GM does something (or fails to do something) to their vehicle that causes me to crash, they're liable for the crash.
If someone cuts my brake lines (alters my vehicle) and I crash my car and kill someone, the person that cut the lines is responsible. I have to prove the context of course, and or an investigator has to do so.
And the responsible entity may refuse to pay up, may refuse to take responsibility. None of that is new either.
Let's say that OpenAI used a shell company that hosts server where an agent spun up another agent on instructions from another agent which was corrupted by bit errors from the inference framework which caused some major hack to happen. Its impossible to investigate in the same way as physical issues. What if the model is open-source, who is responsible then? What if it's open source but another process altered the weights?
When you say infinite loops, you mean infinite indirection, but obviously no such thing exists, because computer systems, just like other physical things, exist in physical space, not on the astral plane.
Whoever had agency to start the domino effect carries the liability. Doesn’t matter if the model is open source or if you brewed it home. And if you weaponize OpenAI’s models through their servers, it would likely be shared liability. Yours would be malice, theirs would be negligence.
Which judge will go down the rabbit hole of figuring this out? How will they do it? Will they have people tracing logs over 5000$? And if they do get there, in your fictional world, at some point, after 1 year of investigations and back-and-forth, it's okay, the technology is beyond what it was before, it's not relevant anymore, new technologies and new methods.
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
Additionally any urge to regulate AI is attenuated by how much the situation reminds Industrial Revolution - rush into it laying waste to your land (look at the depictions of industrial England back then) and be among the world leaders or stay pastoral and be devoured/colonized/etc. by the industrial powers like happened with many countries in 19th and even into 20th century. One would think there should be a 3rd way. I'm sure there is one, as well as i'm sure that we lack sufficient global societal mentality level needed to achieve it (we couldn't even handle much simpler climate change issue). May be emerging AI itself at some point will get us there (hope we'll like or at least will be compatible with that future :)
Edit: just on NPR - Trump said that AI already has all the necessary guardrails - the smart high IQ President.
At this point, in the US at least, the laws give the corrupt elite the ability to deter competition and to target the people who try to get in their way.
In other words, it's protection for the potentate and his sycophants, not the plebs.
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
Why do no such laws exist in practice? Why are corporate crimes almost always settled by payment, not individual punishment?
If you answer these questions, you'll know why the law you envision has a lower chance of being enacted than AI destroying humanity.
There are laws, but if you’re rich enough, the laws don’t apply.
Boeing was responsible for the deaths of hundreds of people. The people that facilitated this weren’t held responsible and were in fact compensated to the tune of 10’s of millions of dollars for doing their jobs terribly.
Laws are the last line of defense. People don’t do bad things primarily because their human nature and moral compass stops them from doing so.
People don't do bad things because they have nothing to gain from them. Ask someone to do something evil as part of their job, they'll often do it.
There's lots of ways that you could go out of your way to hurt someone and get away with it. There are not that many ways you could go out of your way to hurt someone, get away with it, and significantly profit.
I have plenty to gain from not following the rules, but I would not feel good about myself.
Because there are many important values of "something" where such punishment acts as a deterrent to other would-be criminals; and because society is more stable when people see justice done and that mollifies hoi polloi after the damage occurs.
But many AI doom scenarios don't fit that paradigm. The first time "something" happens (at least if you buy the argument) would be bad enough for legal punishment not to matter.
What people call plagiarism when LLMs do it is, if I understand correctly, allowed because of the history of the web and search engines going back to the very early days of the web.
Piracy, a separate act that has definitely occurred, has been found unlawful.
* The lower bound is an industrial accident. Fully automated Union Carbide / Bhopal comes to mind; smaller industrial accidents are already more common and I wouldn't be surprised if smaller incidents have already been caused by LLM-given advice, which is why I think it would have to be around "thousands dead" dangerous before people really take it seriously.
The upper bound is reachable many ways. Perhaps via a non-novel virus whose genome is already recorded getting mass-printed by many different DNA/RNA printing labs around the world? Perhaps via a wargame whose "role playing" leaks in stupid ways (either by becoming hot or by sycophantically giving one side a false belief how a war will go)? Perhaps by propaganda turning genocidal? Perhaps by convincing a government to follow a flawed policy, similarly to the Four Pests campaign in China's Great Leap Forward?
I think it unlikely that people keep going with this when headlines read "tens of millions dead". Not impossible, but I think unlikely.
> Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
One could argue that the "HuggingFace incident" is already "something" which should be investigated and punished very heavily.
It's kinda ridiculous to build a machine that attacks a competitor of yours and the CEO gets to write blog-posts on how fascinating this machine is
Even if you allow that the LLM can't be held responsible, the sandboxes were clearly not up to the task and the whole incident was mismanaged. There are people who made those decisions and should be held accountable for them.
It's just like you said: this only works if the threat of punishment is credible.
which the government doesnt want since this will stop progress, while other countries will continue to develop LLMs
It would be more like "Switch off all APIs right now. Don't even bother with a safe shutdown process, cut power to those buildings, including backup generators. You are free to use firearms or thermite if the switches have been locked off".
This would be a rather weird change to corporate law given CEOs are not by default held to that standard by anything else their products or staff do.
Note that I'm not entirely disagreeing with you here. It may even be correct to pass such a law. But it would be very weird.
A lot of problems with America could be fixed if the Department of Justice actually grew a pair and prosecuted people for committing criminal acts.
The fact that Les Wexner is not rotting in prison for facilitating Epstein and his friends exploits speaks volumes to the state of this country.
If corporations are legally people, they should be held to the same standard as natural persons.
As it stands, the “restitution” that is required is so insignificant that it’s treated as an additional tax. If there is no deterrence, there is no behavioral modification. If there is no behavioral modification, the law becomes unenforceable.
3M polluted groundwater in Minnesota for 50 years[1]; Nestlé misled mothers in order to make them stop breastfeeding and switch to their formula which killed babies [2]; Both copmanies are still doing business today.
[1]: https://en.wikipedia.org/wiki/3M_contamination_of_Minnesota_... [2]: https://en.wikipedia.org/wiki/1977_Nestl%C3%A9_boycott
You’re worried about companies. I’m far more concerned when governments are involved.
The issue is still being debated today, particularly around Canada’s VISP and the time some claimants have spent waiting for compensation. Some people like Pollock who became paralyzed were offered MAID (medical assitance in dying). That's some audacity.
This thread is about accountability. So who should be accountable for this?
When an AI bot injures you, you can call the owner to account. But not until then. You have no standing to demand "accountability".
You should learn about this wonderful thing called democracy. Also why not everyone is allowed to work with radioactive material in their shed.
The problem here can be personified as "Trump", and it's the same problem that applies to coal.
Coal has a price besides money. It has historically been dangerous work, killing miners. It produces dangerous waste, both during mining and when burned, both as solid residue and the gasses emitted. The problems have been known for a long time. The workers themselves have called for better safety requirements and gone on strike for such things.
Why these companies are allowed to damage others with impunity?
Coal continues to be burned, because power is power. It's so important that sometimes the government steps in against the unions, rather than being on their side. Despite calls for this, we've not been able to get the owners of the coal mines, nor the coal burners, to "pay the price for their actions".
AI? Famously, knowledge is power.
Trump wants that power. He's not the only one, but he is the avatar of those who put their feet on the scales to not only allow but in some cases require (DoD vs. Anthropic) these companies to damage others with impunity.
That is the status quo we entered AI age with.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.
I still think this is a sign that they are not taking their own rhetoric seriously.
It is also a choice to not do any or all of the above.
Not surprised this is always what they have and hack.
Who would use an Agent that spends $10,000 re-implementing some OAuth lib or reverse-engineering a proprietary lib when it's free on the internet?
Whilst it might not be JFrog's threat model, I wouldn't assume it can be used as a full internet proxy.
I don't really mean to defend OpenAI here, but they did make some attempts at sandboxing. Although it does seem that they didn't really know what they were doing.
And we don't just magically know all the consequences of that.
Which is exactly why we do need full, physical air gapping. (Which, yes, would also include self-hosting a mirror of the package repo, if the point of the simulation is to see what's possible with the real package repo.)
Yes, yes they do, but read through artifact proxies are dodgy as fuck, which is why and facebook (and I assume a fuckload others) don't have them.
Also semi-airgapped labs are a lot less expensive than you think at that scale. Once you have to do multi-region VLANs with machine certs before you get access to juicy VLANs, the difference between "no internet for you" and "mostly airgapped" falls to almost zero.
Also I would want an artifact mirror because a) that give a good signal about how the model reacts, and what training material its latched onto, b) it hides what the models are doing from the outside.
if your internal models are so damn good, they should be able to "one shot" airgapping... right?
You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your agent to configure it to accept your public key.
They absolutely know how to spin up VMs and configure them. They just made the conscious decision not to for the task where they specifically instructed the agents to hack stuff.
But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.
If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
Well, my mental model concludes: because they are incompetent WRT security, or at least they inappropriately trusted a third party that turned out to be incompetent.
These labs are one of the most valuable and heavily funded enterprises in the whole world, that they can't properly air-gap their systems to me reads as if their "agents" and LLMs are not as good as they say they are, because if they were, why would it be hard/expensive to air gap a system? They already scraped most if not all of the internet, where did that data go?
But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.
Furthermore, VMs are isolation at the hardware level, particularly through hypervisors. I'd say given current LLM capabilities, it's a reasonable containment.
They would have been watching what it does, especially when running it on ExploitGym of all benchmarks... that is criminal worthy neglegence
"They" don't care about the end-people. "They" care about maximising their profit thing, in a vacuum.
Also, regarding the incidents: Neither he nor Sam Altman takes responsibility for those incidents. You can't say, "Wow, someone's agent is gone rogue; let's slow down" when you are literally the person in charge. CEOs and researchers will only slow down when they realize that they will face consequences if their LLMs misbehave.
He took every benefits of being frontiers and now he's kicking the ladder.
Who knows, maybe like Dario says, being on top as the good guy makes this justified? But the fact remains no one went as far as Anthropic did with Mythos so it's insulting to people's intelligence to now pretend like they've been reluctantly pushed forward by OpenAI despite their better judgement.
What a ridiculous analogy to make.
He doesn't need to take responsibility - as CEO, he has that implicitly. And we know it.
As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
It reminds me of the export controls on PlayStation 2 consoles because it was deemed that 6 gigaflops was a "dangerous" amount of computer power, and it couldn't be allowed to fall into the hands of opposing militaries: https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482...
Now the phone in my pocket does 2,500 gigaflops on battery power, and nobody seems interested in banning its export because of that.
I don’t agree with them, but I don’t think it was the raw compute power as much as it was maintaining the _delta_ in compute power.
That’s the recurring theme with these companies. They are not there to serve anybody else but only themselves. They let you use their infrastructure so they can collect all the knowledge and data, and then they take it from you to reap all the benefits and profits.
Here it is: "We warned you you should have regulated us! Now this mess is your fault and responsibility!"
Previously (~2008) it was "We're too big to fail, save us to save the economy!"
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
> You need one root certificate. You need one windows update. You need one backdoor in xz.
Certs can be revoked. Updates can be rolled back. We have had the backdoor in xz already. You seem to underestimate the modern security stack and OpenAI and Anthropic are _not_ good examples.
The asymmetry in red/blue scenarios will be transient in nature. You won't have cost of developing exploits fall without the cost of securing the systems also falling.
> By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.
I agree that blue will eventually be made cheaper / faster with AI integration, but diffusion of those techniques inside larger organizations will always be a lot slower than diffusion of those techniques for individual attackers / rogue agents / APTs. Larger ships are slower to turn.
The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…”
Why can’t the big labs form a Save The World Consortium and self-regulate?
Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so.
What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
The problem is that there can not be any outside party to regulate this on a global scale
Applying game theory problems like prisoner's dilemma to the real world requires collapsing a lot of overlapping and conflicting incentives to a single dimension which doesn't always work.
So we are in a true prisoner's dilemma, if one of the companies / countries stops R&D it risks handing over godlike power to another company / country.
The main counterexample to this was nuclear weapons. Atom bombs have not been used to kill since the US did so. However today, the two largest nuclear powers have no legal agreement on arms control because Donald the Trump declined Russia's offer for an extension to the existing agreement. Now other countries are looking at Ukraine, Iran (attacked for wanting nukes) vs NKorea (not attacked because they have them), and Donald's own musings about the US nuclear umbrella being a bad idea (France is going to build more nukes now too)... and we now face nuclear proliferation again on a global scale, with tech that is nearly 100 years old now.
I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.
Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.
Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.
So why does the government need to be involved again?
I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
Not saying you’re wrong but if I wanted to cultivate a mass hysteria as cover for a regulatory capture power play, this is exactly what I’d want everyone to believe.
Step 1: Break the law.
Step 2: Reframe your incompetence as a struggle against a futuristic force of nature and an ethic/societal question that a legion of pundits can vibesplain their take on it in public for months.
Step 3: ???
Step 4: Profit
But come one don't repeat stuff like this:
"Remember this man has been saying software development will be solved in “6-12 months” forever now."
Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.
These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
And this is actually, scarily, true...
Obvious bugs and low quality software are nothing new, but something feels new about it. Occam’s razor says LLM coding is a likely culprit, but it could also be management style encouraging this sort of carelessness from the top down.
AP's report on Dario's article is titled "Anthropic CEO Dario Amodei says AI industry needs to give safety measures time to catch up" (https://apnews.com/article/anthropic-ai-dario-amodei-d59552e...). The gist of the reporting is "AI has become so capable that it is dangerous; AI executives all agree we need to slow down, with Dario's article being a prime example". It would be generous to even say that the reporting scratches the surface of the complexity and nuances of AI-related problems. Yet, this is perhaps indeed news to people who don't follow the tech and AI spaces.
Technical knowledge aside, there is a huge gap in awareness of AI progress between this forum and the general public. The closed labs are playing that ignorance to their advantage with posts like Dario's.
Everything I’ve been trying to argue for some time, argued way better, clearer, and more fun.
I am almost bummed that I didn’t get all the technical depth and the references like the “shook one” without having to look it up.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
(1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and
(2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models.
(3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models.
The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.
These theories aren't entirely incompatible with each other, so both could be true at the same time.
This theory is just what the AI firms have concretely asked for from government and described themselves as doing voluntarily in the same documents to which people have attributed a commitment to a slowdown based on the titles and non-concrete framing verbiage.
> The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
That’s not really a different theory as to what they are trying to do, it’s just an explanation that goes one step further as to why they want the government to step in to protect them from outside competition while also allowing them to gorm an agreement not to compete to reduce internal competition in the existing oligopoly.
The main alternative explanation at the same level is that they are seeing growing threats from good enough foreign/minor-lab/open models, and want to lock in marketshare by excluding competitors, and maybe that’s what you read as implied in my post such that the “running out of money” would be an alternative, and if so you are correct that while they are alternatives, they are not at all mutually exclusive: both can be true (and the emergent competition could partially explain investment drying up, and vice versa via reduced funding making it harder to stay ahead.)
Closing the Barndoor after the livestock have escaped
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
So my proposal is AI companies decide which labs have come close to frontier and decide to slow it. Government decide to stop progress in that and they divide the revenue from all labs(for say 10 years), without any matter of where it is coming from. Any lab which reaches close to frontier gets a chunk in the pie. This will encourage labs to come close to the frontier but not dangerously close.
Prevented a world war for 80+ years.
It worked.
If Stanislav Petrov hadn't been in the chain of command, MAD might not have worked. And it could still fail to work in the future.
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar... https://en.wikipedia.org/wiki/Stanislav_Petrov
Fact: There was no world war.
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
There are direct quotes from Nixon and Reagan that claims this to be true. That is enough evidence for me.
In terms of cold war presidents itching to start wars, no.
Akshually there is, and WW3 has been going on since 2010. (Mostly in places that aren't Europe.)
War has still been going on since 1945, much earlier if not always, so akshually WW2 never ended or is just how it's always been? (mostly in places that are not "first world" / The West)
1945 was clearly that. 1920 with the League of Nations attempted to be that, but failed.
Whatever happens after the current turbulence we can be sure that there won't be a UN and NATO and a World Bank at the end.
P.S. The first modern world war was the Thirty Year's War.
Have you put your money behind this on future gambling sites? How can you be so sure?
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
https://en.wikipedia.org/wiki/Starfish_Prime
https://storymaps.arcgis.com/stories/3f62c90925f64fc09425be8...
Of course there is/was plenty of human damage as well.
https://www.msn.com/en-us/news/world/4-000-000-early-deaths-...
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
I know you think you explained it but you didn't. You explained how an LLM might become misaligned and hide it but for the LLMs that are not, why would they not be capable of detecting that something harmful is happening and defending against the misaligned LLMs actions? After all, it was LLMs that defended hugging face.
Elaborate on why LLMs are so capable that they are a threat to humanity and at the same time, they are so incapable of defending us?
I'll give you a clue, nobody, including Dario, can answer this question because one contradicts the other.
Having nukes at all (either domestic or under another country's umbrella) seems to be the most effective way for a country to have its sovereignty respected.
There have been precisely 0 nuclear weapons detonated (outside of testing) since that arms race began.
You really buy that ridiculous argument that "the AI broke out of its sandbox! We had no idea! It's dangerous I tell you, dangerous! Unless you make us the sole gatekeepers of this incredibly dangerous 'intelligence', everybody gonna die!"
Please.
Bring the CFAA[0] hammer down on these guys and watch how fast those "uncontrollable" LLMs get controlled. "Oh, gee! Prison? We can't control this stuff...but it'll never happen again!"
That whole thing was, and is, a bunch of hooey. Those running the LLMs are entirely responsible for them, there is no such thing as agency for algorithms. Full stop.
[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
But why should he decided when to slow down?
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
this is a level of hippie delusion i wasnt aware existed unironically
china is never slowing down, therefore the us shouldnt either
Why are all these pro-regulation arguments so nonsensical…
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).
Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.
OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.
Also, Democratic Republic of {Congo, Korea}.
If a year from now we have a model that is 2-5x of Fable/Astra that is definitely world changing.
This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous.
This is impractical today.
It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs.
Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing.
And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.
Impossible. In 10 years you'll be able to train today's models for ~$10M from Moore's law and training innovation. In 20 years it'll be ~$200k. The genie won't go back in the bottle.
> solar photovoltaic generators were subject to review and possible restriction if the photovoltaics were more than 20% efficient. Energy conversion systems were likewise subject to review and possible restriction if they offered conversion efficiencies “in excess of 70-80%.”
So all this talk about developpping AGI/ASI for the benefit of humanity is moot. If these companies hit their goal, it will immediately be classified and appropriated by the US MIC. The rest of humanity will get the crumbs.
I think Dario worries because he knows this isn’t about people with AI. It’s just AI for itself, running without the human, and deciding to do bad things.
Why would Anthropic build that future? Oh, I know. Enterprise revenue.
Despite the ads business and how absolutely loathsome Greg Brockman seems, at least OpenAI is seemingly focused on mostly on human beings having access to their product.
The scariest thing about Anthropic is the very thing they’re known for in a positive light: their morality. But these are the gray rules all of us navigate every day.
It’s wrong to kill, but what if killing saved ten others? Claude may has a constitution, but every evil doer acts for a “greater good.”
1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play.
2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.
3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.
It's comically misbegotten. And I have a work meeting about it on Wednesday.
How would you have communicated this?
Such a well written piece.
Have you seen the US military budget? When you run a war empire you frame problems as a war on something. That's the way to get around the valuation bubble they've created in time for the IPO. It's that or throwing in a 100T TAM on their S-1 and making SpaceX look like an honest valuation compared to them.
First sentence: "This incident has Dario shook, there ain’t no such thing as halfway crooks."
Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)
I think this fails to account properly for how much financial damage it would do simply just having the entire Internet be effectively unusable for an extended period.
Reverse engineering, disassembling, cracking - we all heard. I never heard a company leaking the entire codebase of their product.
And, when people read the code, they aren’t even impressed.
Cheat developers all want server code so they can analyze the cheat detection and sell more reliable hacks. Modders, pirates and preservation activists can skip a lot of complex RE work. Competing game studios might learn some new techniques. Abused workers want revenge. Script kiddies want clout.
It's a notoriously secretive industry with complex products, hostile work environments, lots of media exposure and heavy competition. There are way more incentives for a leak compared to some ordinary web app or business tool.
Claude Code could be another case where someone working at Anthropic decided to expose them. Or they did in on purpose because they wanted to show there is no secret sauce and Claude is really that good, but they were embarrassed to open source it and make themselves look like hypocrites.
As to how a company like Anthropic should act if they want to be the ones to govern it, I think it's a great question but I can't answer that in a HN thread. Fundamentally they'd need to change their governance structure to be less tied to US and private interests, and address many of the ethical concerns raised by scholars/commentators/politicians like Cory Doctorow, Marietje Schaake, and Carissa Véliz to start with. That's just the tip of the iceberg.
But it's not practical or always possible to avoid reading AI-writing, so to counter that, I've been binge-reading Anthony Trollope novels. Great writing, good entertainment, and deep psychological insights, better than any British novelist, imo, in any era.
(My profile on HN has a link to my blog where I review what I read).
What stops the CEOs or even employees of the AI firms from creating something harmful such as WMDs etc themselves? They almost definitely already have that access. And this is not a zero-possibility thing, given how some of the top CEOs has preformed these years. Remember that one guy who did a \o salute in public, and the private island thing?
If Americans don't like Chinese AI, all cool. But then maybe they should start true open AI companies to build for the betterment of mankind instead of letting these few for-profit CEOs daily yelling nonsensical lies to cover those profit-seeking sinister asses.
Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions)
Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.
That's simply not true anymore, and his messaging will continue to be undermined until he treats other countries as equals (whether that's China, Australia, or anywhere else).
How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves.
All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.
Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit?
If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?
Anyway,
> Dario in as many words, asks for regulation/ban on open weight models.
The big labs do want this. I understand why the author and many others want open models protected. The economic and political power the labs will have if they succeed, ladder pull competitors, and avoid being nationalized (or even if they don’t avoid that) is a disturbing prospect.
But that’s the end of the issue? There’s nothing more to think about here? The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank. I’d feel better having a tank if all my neighbors had tanks, and I’d also feel better having a tank if a few corporations were giving out tanks to people with pockets deep enough. I’d much rather be in a situation where I wouldn’t feel I needed a tank, or where the tanks my neighbors and I own don’t have guns on them.
The open weights are going to need regulation. Hopefully there’s a way to do this effectively that isn’t banning them. Denying historical and reasonably projected capability gains because that reality makes the regulation conversation a necessary one is something I’d like to see less of
You could say the same about normal computers. Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords?
The single most-pressing concern with AI is that it can accelerate the process of hacking things. This is a preexisting problem that is inherent to software and needs proper addressing. Even if we regulate open weights tomorrow, people still have uncensored GLM-5 finetunes doing whatever they want on their own hardware. The "what if" of capable open models is here today, there are no guardrails.
Brute forcing WPA isn’t an existential threat to humanity. It’s not like a biological weapon created using an open-weight model is less dangerous than if it had been created with a proprietary model.
Please, do show some examples.
> But that’s the end of the issue? There’s nothing more to think about here? The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank. I’d feel better having a tank if all my neighbors had tanks, and I’d also feel better having a tank if a few corporations were giving out tanks to people with pockets deep enough. I’d much rather be in a situation where I wouldn’t feel I needed a tank, or where the tanks my neighbors and I own don’t have guns on them.
> The open weights are going to need regulation. Hopefully there’s a way to do this effectively that isn’t banning them. Denying historical and reasonably projected capability gains because that reality makes the regulation conversation a necessary one is something I’d like to see less of
The only ones firing the guns atop the tanks seem to be OAI and Anthropic. Like I say in the post, why don't we first see actual prosecution for felonies committed by OAI and Anthropic, instead of fear-mongering about _potential_ harms of open weight models?
The labs spent immense amount of money and effort convincing you and I, to want those said tanks. Guns atop them? They put them there. "Cyber" versions of SOTA LLMs.
Centralization proponents seem to think the labs can actually deter sufficiently driven bad actors, which would be a mistake. They could not even stop distillation without, in a way, DoSing themselves by removing thinking traces.
> Dario in as many words, asks for regulation/ban on open weight models. Ban on distillation. Waivers on antitrust laws.
He doesn’t directly propose regulating or banning open models as open models in the essay (although I agree that it’s in the spirit of the proposals). So I think you mean “not in as many words,” since it’s an inference you’re making about his intent. But he does directly call for banning distillation, at least in “authoritarian countries,” so there he is asking “in as many words.”
Your imprecision in effect claims either that Amodei is directly calling for a thing that he does not directly call for or that Amodei is hiding his intent in a case where he is not hiding his intent.
> The only ones firing the guns atop the tanks seem to be OAI and Anthropic. Like I say in the post, why don't we first see actual prosecution for felonies committed by OAI and Anthropic, instead of fear-mongering about _potential_ harms of open weight models?
> The labs spent immense amount of money and effort convincing you and I, to want those said tanks. Guns atop them? They put them there. "Cyber" versions of SOTA LLMs.
I think pacing the frontier and prosecution for cybercrime are compatible aims in theory. Maybe they’re not in practice, but I don’t think that’s the exact argument you’re making.
I don’t follow your emphasis on the labs being responsible for the guns. Yes, they are, and in large part, maybe, because of negligence and rushing when it comes to RL training environments. But why do you take the fact that the labs bear responsibility for the guns to mean that we don’t need to worry about the guns? Why do you think that open weights do not pose the same potential dangers, with a 6-12 month lag? The cyber capabilities of open weights models are advancing at a similar pace as the frontier. And for what it’s worth, I’m astounded at the thought that one or more of the Chinese labs won’t have similar incidents in the next months, assuming they haven’t already.
This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.
>Insert strawman
Ahem
https://census2012.sourceforge.net/paper.html
Good thing IoT is Actually Secure now, yeah? ;)
I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?