Comment by SonOfLilit | Hacker News Reader

Comment by SonOfLilit | Hacker News Reader

I wondered if the article was over-dramatizing what risks were covered by the bill, so I read the text:

(g) (1) “Critical harm” means any of the following harms caused or materially enabled by a covered model or covered model derivative:

(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.

(B) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting, a cyberattack or series of cyberattacks on critical infrastructure.

(C) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:

(i) Acts with limited human oversight, intervention, or supervision.

(ii) Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.

(D) Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.

(2) “Critical harm” does not include any of the following:

(A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

(B) Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.

(C) Harms that are not caused or materially enabled by the developer’s creation, storage, use, or release of a covered model or covered model derivative.

Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

In that case, how much of an enabler is an AI to meet the destructive ends, when, if the humans can use AI to conduct the damage, they can surely do it without the AI as well.

The potential for destruction exists either way but is the concern that AI makes this more accessible and effective? What's the boogeyman? I don't think these models have private information regarding infrastructure and systems that could be exploited.

“Critical harm” does not include any of the following: (A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

The bogeyman is not these models, it's future agentic autonomous ones, if and when they can hack major infrastructure or build nukes. The quoted text is very very clear on that.

Ah, thank you, skipped over that part.

I am not convinced the text means what you say it means.

All knowledge (publicly available and not) and all tools (AI or not) can be used by people in material ways to commit the aforementioned atrocities, but only the models producing novel knowledge would be at risk. I hope you can see how this law would stifle AI advancement. The boundary between what's acceptable and not would be drawn at generating novel, publicly unavailable information; not at information that could be used to harm - because all information can be used that way.

What if AI solves fusion and countries around the world start building fusion weapons of mass destruction? What if it solves personalized gene therapy and armed forces worldwide develop weapons that selectively kill their wartime foes? Should we not have split the atom just because that power was co-opted for evil means, or should we have not done the contraception research just because the third Reich used it for sterilization in their war crimes? This bill would work towards making AI never invent any of the novel things, simply out of fear that they will be corrupted by people as they have been in history. It would only slow research and whenever the (slower) research makes its discoveries, they would still get corrupted. In other words, there would be no change in human propensity to hurt others with knowledge, simply less knowledge.

Besides, the text is not "very very clear" on AI if and when it hacks major infrastructure or builds nukes. If it was "very very clear" on that, that is what it would say :) - "an AI model is prohibited to be the decision-making agent, solely instigating critical harm to humans". But what the text says is different.

I agree that AI harms to people and humanity need to be minimized but this bill is a miss rather than a hit and the veto is good news. We know AI alignment is needed. Other bills will come.

I'm pretty sure there's a few hundred fusion wmds in silos a few hours north of me, we've had this kind of weapon since 1952.

Nice fact check, thank you. I didn’t know H-bombs used fusion but it makes complete sense. Hydrogen is not exactly the heaviest of elements :)

Well then, for my example, imagine a different future discovery that could be abused. Let’s say biological robots or a some new source of useful energy that is misused. Warring humans find ways to corrupt many scientific achievements for evil.

Sharks with frikkin lasers.

If a model were capable of novel WMD research, I would want it to stay out of reach for the general public! Wouldn't you?

After we split the atom, and saw how dangerous it was, we established some damn strong protections in place to prevent kids from building nukes in their backyards for giggles. If we ever build a physicist AI that is better than Einstein at WMD-relevant research, I sure as hell hope we don't just let the kids ask it how to build nukes or antimatter bombs or whatever. And this is exactly what this regulation is about.

> But what the text says is different.

Since "prohibited" is a word that carries specific consequences, and lawyers are all about consequences, legal texts usually phrase it something like "if a model does critical harm to humans, the human responsible for building and releasing the model will be culpable for the crime", which is a very good TL;DR of the relevant parts of the bill. Isn't it?

Newsom is the governor who vetoed the bill, not the lawmaker who authored it.

> Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

AI models might not be able to, but an AI system that uses a powerful model might be able to cause damage (including extinction of humanity in the more distant future) unintended and unforeseen by its creators.

The more complex and unpredictable the system the harder it is to properly oversee.

--- (2) “Critical harm” does not include any of the following:

(A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

---

This exception swallows any rule and fails to target the difference with AI: it's actually better than an ordinary person at assimilating multiple fact streams.

That suggests this law is legislative theater: something designed to enlist interest and donations, i.e., to build a political franchise. That could be why it targets only the largest models, affecting only the biggest players, who have the most resources to donate per decision and the least goodwill to burn with opposition.

Regulating AI would be a very difficult legislative/administrative task, on the order of a new tax code in its complexity. But it will be impossible if treated as a political franchise.

As for self-regulation, with OpenAI's changing to for-profit, the non-profit form is insufficient to maintain a public benefit focus. Permitting this conversion is on par with the 1990's+ conversion of nonprofit hospital systems to for-profit.

AI's potential shines a bright light on our weakness in governance. While weak governance affords more opportunities, escaping the exploitation caused by governance failures is the engine of autocracy, and autocracy consumes property along with all other rights.

(I added newlines to your quote to match what looked like the intended formatting - I hope that's ok. Since HN doesn't do indentation I'm not sure it helps all that much...)

I’m sure people have asked this before, but would HN ever add a little more rich-text? Even just bullet points and indents might be nice.

And maybe also make new lines in the comment box translate to new lines in the resulting comment...? :D

That's actually a really good point. I've never looked into that, I just took it for granted that to get a line break on HN you need two consecutive newline chars.

I guess the thing to do would be to look at all (well, lots of) comments that have single newlines and see what would break if they were rendered as actual newlines.

Could be applied to all comments made after a certain future date. That way nothing in the past is poorly formatted

Or just brute-force it with string_replace of all "\n" with "</p>\n<p>" and then remove all the empty "<p></p>".

(why yes, i am a PHP guy, why do you ask?)

Maybe. I'm paranoid about the unintended cost of improvements, but it's not an absolute position.

Reply on news.ycombinator.com