Hacker News Reader

Comment by drillsteps5 | Hacker News Reader

These companies are building software. That doesn't work very well. The output it produces does make sense at times, but there are times when it doesn't. And instead of fixing that, or admitting it can't fixed, they started bolting actuators to them, executing actions online (for now) based on the output of their buggy software.

And when this results in actuators executing some bad actions they scream in horror "AI went rogue! It escaped the containment!!! It's going to kill us all!!!"

Go fix your software before you let it do stuff online or IRL. It's not "Terminator", it's just bad QC.

View on HN

But the thing is, they are going to keep bolting more and more actuators on, and training more and more powerful agents, and we (society, especially the tech industry) are going to keep using them, because they are extremely useful. And I don't see why you're so confident that frontier agents can't get powerful enough to do serious, real, lasting damage to the world; as far as I can tell, AI models have been improving at an accelerating rate, and there is no sign that that is slowing down or will slow down in the near future.

You're missing the point here.

If we take the major LLM companies' claims at face value, they're knowingly building WMDs that have a high probably of wiping out the entire human race. [0] Manufacturers that are designing, building, and selling that sort of thing need to have a dreadfully serious culture of safety.

When manufacturers run live tests of their extremely dangerous -again, the claim of danger is their claim- tools with the tools' safeties removed, one expects that those tests will be run on a carefully-controlled range cleared of all bystanders. One also expects that the results of those tests will be scrutinized and everything that got damaged that they didn't intend to be damaged will be noticed and noted very quickly after the conclusion of the test.

In actuality, these manufacturers connected said tools to the Internet and did not discover the unintended damage caused by those tools until weeks to months after the tests. In some (most?) cases, they had to be notified of the damage by the damaged party! This means that their safety culture is entirely inadequate for the dangerous task they've deliberately chosen to undertake.

[0] A 10% chance of causing the destruction of the entire human race is -given the stakes- _enormous_.

It seems that we’re mostly in agreement? I agree that OpenAI has been terribly irresponsible, and that this attack being an accident makes things worse. I think we need strong action now to stop the frontier companies from developing dangerous, powerful AI agents that they don’t know how to control.

> I think we need strong action now...

You and I and Nvidia CEO Jensen Huang seem to agree on this. Excerpts from his interview with Ezra Klein: [0]

Klein:

  But what I hear the various people in the lab saying is: We are in this. We feel we are losing control of what we are creating. We want help to slow down where it’s not a collective action problem.
  
  So why are you resistant to that?

Huang:

  Because these are companies with agency. These are C.E.O.s with agency. ... They could absolutely take care of the situation.
  
  Ezra, it’s so weird. If a car company, competing with a bunch of other car companies, which they are — I’m competing with all kinds of companies, which I am. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability, my power and my responsibility, and I’m incentivized to do so, to not launch the product.
  
  And so I can’t buy into the idea that somehow, all of Americans, around 400 million of us, are pushing them to launch untested products that are unreliable, engineered poorly, because they thought they were trying to help us. Don’t do it for me, OK?
  
  And therefore, I think we’ve got to break it down. I mean, it’s really, really serious.
  
  The fact of the matter is, there are so many laws, there are so many obligations, they’re so incentivized to ship safe products. If they ship unsafe products, their customers go away. If they ship unsafe products and they harm somebody, they could have a civil lawsuit. If they ship something and they did it knowingly, there could be negligence involved. There could be criminal lawsuits.
  
  The fact of the matter is, there are plenty of incentives for them to do it right. So I have to disagree with your premise that somehow somebody’s pushing them to do this. Nobody’s pushing them to do this. ... I’m saying that we have lots of laws and regulations. Apply it.

Former FTC chair Lina Kahn has suggestions, too. [1]

Thoughts?

[0] <https://www.nytimes.com/2026/09/23/opinion/ezra-klein-podcas...>

[1] <https://x.com/linamkhan/status/2099204390548639960>

I agree that the weight of the law should be brought to bear on OpenAI. Ideally they should be brought in to testify before congress as well. I'm not a lawyer so I can't really comment on whether current laws are sufficient, but I believe that there should be laws specifically governing the development of AI, with a requirement that a given architecture and reinforcement mechanism be _proven safe_ before training begins.

> ...I believe that there should be laws specifically governing the development of AI...

Why? Existing truth-in-advertising, liability, safety, and -where and when appropriate- weapons-development laws and regulations constrain the past and current conduct of the LLM manufacturers just fine.

The only possible reason for making new laws that I can see [0] is that existing laws "don't work" because the LLM manufacturers are ignoring them. Which, like, _if_ the new laws are going to actually constrain their behavior, why the hell would the LLM manufacturers pay any attention to them? They've already demonstrated that they give zero shits about the existing laws that prohibit what they have been doing and continue to do.

[0] ...that isn't "The LLM manufacturers are engineering a panic with their very real, actual, and actually alarming conduct so that they can 'guide' lawmakers and regulators into 'accidentally' letting the LLM manufactures capture those who would regulate their behavior"...

Liability only kicks in after damage has been done. As far as I know, there is no law that could currently force AI companies to only test cybersecurity capabilities in air-gapped datacenters, for example; only laws that could punish them if their cyber testing led to a hack that caused material damage. But if, lets say, a rogue AI agent swarm attacked a hospital and caused patients to die, no amount of liability will bring those patients back to life.

> Liability only kicks in after damage has been done.

a) Both OpenAI and Anthropic have done far more damage with their jaw-droppingly-sloppy testing of computer-attacking tools than Aaron Swartz did by downloading documents from JSTOR. It's good to see that you and I both agree that there are things for them to be prosecuted for.

b) Is your claim that the cost to thoroughly investigate and clean up after a cyberattack doesn't count as damage? If so, that runs contrary to every relevant claim of damages in a CFAA case that I've seen.

No I absolutely think they should be prosecuted, and at a minimum owe damages to all of the companies that their agents hacked.

What I'm saying is that that is not sufficient to stop future harm; I expect the total damages would be less than the cost of a full training run, so it would effectively just be the cost of doing business. Liability is not sufficient to protect the world from dangerous technology - we need proactive rules around how the technology is developed, tested, monitored, and deployed, as we do with other dangerous industries such as airplanes, nuclear reactors, weapons manufacturers, etc

> Liability is not sufficient to protect the world from dangerous technology - we need [new] proactive rules...

You and I couldn't disagree more.

The major LLM manufacturers are begging for new laws and regulations so that they get a huge hand in writing them. Regulatory capture is absolutely their goal. Given that they claim to believe that they're working on WMDs [0] that they cannot adequately control, they'd just stop work if safety was their goal. Their collective cries for regulation demonstrate that they'll happily coordinate with each other if they think the issue is important enough to do so. I guess "preventing the extinction of the human race by way of weapons we built and let slip from our hands" isn't sufficiently important.

[0] See the second paragraph and associated footnote here for a justification for the use of this term: <https://news.ycombinator.com/item?id=49839682>

Reply on news.ycombinator.com