Cali's AG Tells AI Companies Almost Everything They're Doing Might Be Illegal | Hacker News Reader

Cali's AG Tells AI Companies Almost Everything They're Doing Might Be Illegal

gizmodo.com

181 pointsclumsysmurf153 comments

What is being argued in the article that AI Companies could be doing illegally:

-Using AI to “foster or advance deception.”

-Falsely advertising “the accuracy, quality, or utility of AI systems.”

-Create or sell an AI system or product that has “an adverse or disproportionate impact on members of a protected class, or create, reinforce, or perpetuate discrimination or segregation of members of a protected class.“

What kind of standard is disproportionate impact?

Do not build an AI system that discovers the best way to target men to get a prostate exam, or the elderly to enroll in a program that may benefit them?

I understand "adverse", but disproportionate implies don't even try to help classes of people. AT ALL!

What they are talking about is using AI as a proxy for racial, gender, and age discrimination. For instance, if you use AI for an adverse decision regarding employment, housing, education, or financial decisions you need to justify those decisions against persons from a protected class.

If you cannot explain why you have made such decisions, then the state will look at disproportionate impact. EG: You hired/lent to/rented to black people 25% less often than you did to white people. "Because the AI said so." doesn't cut it.

Your argument makes it sound like the statement says "adverse and disproportionate". I'm seeing that it says OR, meaning you don't have to do anything adverse.

I don't like when laws are worded such that we told we shouldn't care what the words say, we all know what we meant, this is a law for getting the bad guys, so don't worry about the actual words we use. You shouldn't need a law degree to know what OR means.

Don’t make the mistake of confusing colloquial language with legal text. They do need a law degree, or at least knowledge of the precedent that precisely defined the term. Precisely defining them is half the battle.

“Adverse or disproportionate impact” is a well litigated phrase. It has a specific meaning in law, which is not immediately obvious from a layman’s definition.

Legal writing brings in a lot of context. In this case the disproportionate is meant to denote situations where the action itself isn't adverse, e.g., giving someone money, but that its not applied evenly. Whereas adverse is an intended negative consequence. Its trying to say both are problematic. Although, I understand your interpretation -- its just not what's meant in this case. Unfortunately even arguably our most important legal document, the Constitution, isn't nearly as clear as it could be.

Typically wording will either be defined in the law, or decided by case law. Legislatures will usually include very long sections on definitions to try and reduce the amount of judicial leeway.

For instance, in Texas motor vehicle code regarding driving under the influence, there is a definition for "Motor Vehicle." While colloquially a person would assume that such a vehicle should have a motor, the definition actually states:

> "Motor vehicle" means a device in, on, or by which a person or property is or may be transported or drawn on a highway, except a device used exclusively on stationary rails or tracks.

When it comes to laws, you have to read the entire law (including definitions) and not just rely on your own understanding of the terms within. Then it gets even more complicated when it comes to so called "Case Law." This is why companies have entire sections of lawyers to inform their managers on compliance.

That's a great example. I've seen in other states that the definition of "motor vehicle" encompasses bicycles. It probably is the simplest way to ensure that all traffic laws automatically apply to bicycles, and then some special cases for bicycles can be defined separately.

I interviewed for a consulting role circa 2019 where the company audited models of finanical companies that used their models to make lending decisions.

A large portion of the interview touched statistical inference as it related to ML, specifically how it related to simple neural nets up to deep learning vs classical modeling. The answer I gave aligned with their expectations, which was that models used in lending should not be black box and should be able to quantify which features led to the prediction/output and how much weight they contributed. This was specifically done to address potential discrimination lawsuits.

I have a hard time believing any company that rents, lends, etc. would employ a black box for decisioning. Both private and public lawyers would sue them into oblivion immediately.

there is an entire specialty of Model Risk Management, banks have entire departments that oversee models used for decision making and ensure compliance with fair lending laws and other laws

https://www.occ.treas.gov/publications-and-resources/publica...

One prominent counterexample is that made famous after the UnitedHealthcare CEO was shot and killed, that they employed a decision-making algorithm that apparently rejected 90% of claims with no audit trail or real justification besides cost estimates.

That is an unsubstantiated (so far) claim by plaintiffs in an active lawsuit. I would not consider it data for any example yet.

https://www.theguardian.com/us-news/2025/jan/25/health-insur...

> What they are talking about is using AI as a proxy for racial, gender, and age discrimination.

See also: "It's not a crime if you do it with an app" [/s] - https://pluralistic.net/2025/01/25/potatotrac/

I do work in recruiting. If you let AI making hiring decisions for you, you are an idiot. All it takes is one disgruntled applicant to complain about discrimination in your hiring process to cause you a world of pain. You need to be capable of owning your hiring decisions, meaning you can’t just point to the AI and say “it told me not to hire this person”. You won’t be able to avoid liability when it turns out your AI is biased against some protected class.

I think the place people get into the most trouble with hiring is when they use AI to filter applications. It's one thing to filter based on a specific criteria like education or experience, but to go by "AI didn't like the look of this resume" is something else.

What AI systems are currently targeting my prostate?

This space is primarily dominated by AI that replaces mundane work done by humans.

The AG's advisory is essentially a memo from the AG explaining how the AG thinks existing statute applies to a new technology. It's not new legislation or new case law, but it is worth paying attention to for various reasons.

The section of the advisory referencing disproportionate impact is quoting, nearly word for word, a portion of Cal. Code Regs. Tit. 2, § 14027.

So, this section of the advisory essentially amounts to the AG saying "using AI to do something illegal is still illegal".

That does not really answer your question, though.

> What kind of standard is disproportionate impact?...

The kind that does has been codified in CA and other jurisdictions' statutes for a long while now. This means that the standard is extremely well-litigated in the state's courts, and so the answer to "what is disproportionate impact?" is, I think, something like:

"That seems complicated; there's probably a rich case law that provides clarity in some situations but also highlights areas of ambiguity in other situations. If you're in it for profit, and have any questions, get a lawyer who specializes in that area of the law to review your specific circumstance; if you're in it for civics/curiosity, start with the statute then start reading significant case law or law reviews regarding that statute."

It's also the kind of standard that can attract flame wars... hopefully not here, though ;-)

Men aren’t a protected class anyway.

For the last point this made me think squarely at chatbots. I had a chatbox experience recently that could have been handled with a cancel order button on a website. I had to go through a phone tree with the chatbot, say “no” 3 times until it let me type a free response. Said I wanted to cancel an order. “Did you mean cancel order?” Yes. Yes I did. What should have been a 2 second interaction with a ui element became a 5 minute song and dance. And I am a reasonably competent person on a computer. For someone who isn’t maybe this is enough to make them turn away from whatever action they were attempting to do in frustration. That is a big impact. And depending on where these chatbots have been shoehorned in, that impact could easily be disproportionately burdened on a protected class.

> I had to go through a phone tree with the chatbot, say “no” 3 times until it let me type a free response.

I’m not excusing this design (it could probably be improved), but it may have been intentional.

See below.

> And I am a reasonably competent person on a computer.

Most people are not. It’s generally unwise to design an automated system that assumes computer/tech competence.

> For someone who isn’t maybe this is enough to make them turn away from whatever action they were attempting to do in frustration.

I imagine it’s the other way around. This type of system saves folks with less tech savvy from themselves.

I’m not sure if you’ve designed systems like this before. I have, and I was very surprised at what people thought was reasonable interaction and/or reasonable input.

Confirming choices, perhaps multiple times, before moving forward can save a lot of headache later for everyone involved. The collateral damage is guaranteed “wasted” time for everyone using the chatbot, but the company largely doesn’t care about that — that is, they are more than willing to pass on the cost of their money (e.g., if they hired a human agent) for your time.

We are talking about a cancel button here. If you can wrap your head around the order button they also have, you can wrap your head around a cancel order button. A phone tree is stupid lazy ux. Even worse when you now have to massage it out of a chatbot which isn’t even reliable in how it responds back to you like a phone tree used to be.

So, once again, everything is securities fraud?

- “Foster or advance deception” - definitely happening. Many (maybe most on some platforms) posts on social media are being generated by LLM-type AI systems. The question of whether a power tool for crime is aiding and abetting a crime is a tough one. The gun industry beat that. It has a lot to do with how the product is promoted.

- "Falsely advertising the accuracy, quality, or utility of AI systems” - worst case is probably Tesla's Fake Self Driving. Most other LLM systems aren't allowed to make decisions, just blither.

- "Create or sell an AI system or product that has an adverse or disproportionate impact on members of a protected class, or create, reinforce, or perpetuate discrimination or segregation of members of a protected class.“ - hm. Need more instances. Now, using computer systems to create a cartel to violate antitrust laws and push prices up is a thing. There's litigation against landlords for that. But that's not a "protected class" thing, it's an antitrust thing.

not to seem pedantic but the self driving system isn't an LLM, it's probably using an artificial neural network for some aspects like object recognition (and would have similar issues like need large amounts of data) , but the similarity ends there.

Waymo's isn't an LLM. Tesla's is supposedly a pure machine learning model.

Even though he’s right, Rob Bonta is going to get himself fired, while Scott Wiener will write a bill legalizing training on non expressly licensed data.

Fun fact: Germany's IP law has a provision that allows AI training by default for everything that's reachable on the Internet, if the website operator hasn't published a "nope" in machine-readable form (i.e. robots.txt).

[1] https://www.gesetze-im-internet.de/urhg/__44b.html

So to legally copy a website, all you need is to just pass it through an AI filter and then you can legally publish the rip off?

No - data mining copyrighted material and republishing copyrighted material under your own name are two very different things

Passing data through an AI filter for "training" is a different thing (legally and ethically) from publishing the output.

If you read the law it says it’s only allowed if the rights holder doesn’t disallow it (in machine readable form) - I would argue robots.txt falls under machine readable

>will write a bill legalizing training on non expressly licensed data.

Which should be assumed to be legal already, even without the expressly written bill. Copyright maximalism is anti-human.

This is the way. We need to restrict IP protection, especially temporally.

Stealing is more anti human.

Surprisingly when it comes to software, music, and movie stealing we find that stealing requires one party to lose something but when it comes to OpenAI stealing is happily colloquially defined. What an interesting curiosity.

When I steal a thing from you, you no longer have the thing.

When I steal a dance you just invented, you're very butthurt about it and run crying to mommy "make him stop copying me!". Then you grow up and bribe Congress to make it illegal. Except for the "growing up" part, that never happened.

Expecting megacorporations to play by the same rules they want us to follow when in comes to their rights is pretty far from copyright maximalism. Anti-human is giving corporations more rights than humans.

> "The law, in its majestic equality, permits rich and poor alike to massively-plagiarize anything they want after investing at least $100,000,000 on a computational pipeline to statistically launder its origins and details."

-- Cyberpunk Anatole France

____

If I were to steel-man your comment, it would be something like: "Scraping and training must be fair-use because people can be building all sorts of systems with ethical and valuable purposes. What you generate from a trained system can easily infringe, but that's a separate thing."

Also, where does the GNU Public License fall in terms of "anti-human copyright maximalization"? Is it bad because it uses fire, or is it good because it fights fire with fire?

>it would be something like: "Scraping and training must be fair-use because

It wouldn't be "fair use". It makes no copies. "Fair use" is the horseshit the courts dreamt up so they could pretend copyright wasn't broken when a copy absolutely needed to be made.

This makes no copies, so it doesn't even need "fair use". Instead, there are people who believe that because they made something long ago that they and their descendants into the far future are entitled to tax everyone who might ever come across that thing let alone actually want copies of the thing.

Your argument must sound intelligent to you, but it starts from a premise of "of course copyright is the only non-lunatic policy people could ever imagine", and goes from there. You can't even think in any other terms.

> Also, where does the GNU Public License fall in terms of "anti-human copyright maximalization"? Is it bad because it uses fire, or is it good because it fights fire with fire?

Stallman is clever to twist the rules a little to get a comparatively sane result from them, but there are others who aren't clever enough to even recognize that that's what he's doing. So, in their minds "what about the gnu license" seems like a gotcha. I won't name those people, but their username starts with Terr and ends with an underscore.

> Your argument must sound intelligent to you, but [...] You can't even think in any other terms.

> others who aren't clever enough [...] I won't name those people, but their username starts with Terr and ends with an underscore.

https://news.ycombinator.com/newsguidelines.html

____________

> It wouldn't be "fair use". It makes no copies.

Incorrect, the real-world behavior we're discussing involves unambiguous copies, where LLM companies scrape and retain the data in a huge training corpus, since they want to train a new iteration of the model when they adjust the algorithms.

That accumulation is analogous to photocopying books and magazines that you borrow/buy before returning/selling them again, and arranging your new copies into a clubhouse or company break-room. Such a thing is not usually considered "fair use."

In a hypothetical world where all content is merely streamed into a model, then the question of whether model-weights can be considered a copy with a special form of lossy compression is... separate, and much trickier.

> Your argument [...] starts from a premise of "of course copyright is the only non-lunatic policy people could ever imagine"

Nope, it's just the context of the discussion because it's status-quo we're living with and the one we're faced with incrementally changing. If you're going to rage-post about it, at least stop and direct that rage appropriately.

> Stallman is clever to twist the rules a little to get a comparatively sane result from them, but [you don't] recognize that that's what he's doing.

I already described the GPL as "fighting fire with fire", I don't understand how the idiom didn't make sense to you.

I don't think the memo mentions training data sources; it's about usage and impact.

>Rob Bonta is going to get himself fired

The Attorney General of California is an elected position. He could be recalled but not fired by the Governor.

A recall is firing.

Sort of? But it is very different to the federal cabinet where they serve at the pleasure of the president. The recall process is slow, expensive, and rarely successful.

>Since 1913, there have been 181 recall attempts of state elected officials in California. Eleven recall efforts collected enough signatures to qualify for the ballot and of those, the elected official was recalled in six instances.

https://www.sos.ca.gov/elections/recalls/recall-history-cali...

But a firing by the electorate, not a firing by the governor of CA.

Sure, but it's still getting fired. Also, the governor can support the recall (And Gavin should).

I'm not sure how... likely that is? Like, is the California electorate particularly enamoured of the LLM-flogging mega-corps, such that they would do their bidding in this way?

Like, if the reasoning is "we should recall them because they were mean to the lovely companies :(" then I'd expect the average person to say, broadly, "good" and vote against recall. 'AI' is not particularly popular with the public.

He co-authored Senate Bill 239, which lowered the penalty of exposing someone to HIV without their knowledge and consent from a felony to a misdemeanor

That Scott Wiener? How does he still have a job?

Everyone qualified to speak on such things was pretty universally in agreement that the previous law was increasing the spread of HIV rather than decreasing it, as the primary effect it had was that sex workers would refuse to get tested.

He’s more likely to get rounded up with Sheng Thao, Andre Jones, Bryan Azevedo, and his wife, Mia Bonta.

> Use AI to foster or advance deception. For example, the creation of deepfakes, chatbots, and voice clones that appear to represent people, events, and utterances that never existed or occurred would likely be deceptive.4

> Likewise, in many contexts it would likely be deceptive to fail to disclose that AI has been used to create a piece of media.

Under this logic a company making pencils is illegal.

Could you please develop your idea? I don't see the link between pencils and deepfakes.

I can write a letter for propaganda purposes already. I can draw a picture of someone and say "This is the president doing a nefarious act!". They are confusing a tool possibly being used for a purpose as making the tool illegal and not the purpose. If writing propaganda via AI makes AI illegal, then we should ban pencils since they can be used for write propaganda. Pencils can be used to forge documents, but again they are legal.

Firstly, do note that the original text is talking about use of AI, this one line doesn't seem to be about banning AI so much as banning a use.

Secondly:

The implied basis here is that AI isn't just a product, it's also a service. This isn't you buying a pencil, it's you commissioning the drawing. Most of these products are cloud based SaaS.

And there's also the matter that "it's just a tool" doesn't really apply to foreseeable problems. If a suspicious person shows up out of nowhere buying large quantities of fertilizer, you don't get to go "Well he could be using that fertilizer for anything, not my problem". (This is relevant to AI as pretty much all AI services already have heavy restrictions on their output, this isn't a bunch of researchers publishing a paper and having bad actors implement their own AI based on that. We have companies openly advertising deepfake services.)

You can create all kinds of nefarious artworks that are hurtful to protected classes using a pencil, or adobe creative suite or what have you. Doesn't mean those things are illegal.

Charge the creator not the tool. This seemed immediately obvious to me, but apparently not to most layman (including the California AG, apparently).

Do you have a magic pencil that can do all this without human involvement?

We're splitting hairs here. Open up the ChatGPT window and have it do something nefarious without you touching the keyboard. Then these AG actions would make sense.

It's not splitting hairs at all. If I ask you to do something illegal and you do it, who do you think is liable?

Does it matter? A few years ago I had the same concern, but it's been easy for a while now to create photorealistic fakes of anything you'd like, and in practice it doesn't seem to have been more convincing than any other form of lying. There are of course a couple specific areas where the mere existence of a picture matters even if people know it's fake (see e.g. https://www.edweek.org/leadership/students-are-sharing-sexua...).

I am guessing their point is that pencils can be used to create deceptive content.

Beyond that, I don't see any similarity. In a pencil, all of the "intelligence" is offloaded onto the user. With AI, the company providing the service is playing a more substantial role.

All governments now think that if data is streaming through you as a company (anything cloud) then you have the LEGAL REQUIREMENT to execute all their arbitrary censorship for them.

AI is a brand new frontier where they can even pre-program their arbitrary censorship into things you download too! For as much offline censorship as possible too. Wow innovation

We have to do peer to peer, and avoid cloud anywhere and everything like we were asleep for 10 years

If these companies were just putting their models up as Open Source, then your "pencil manufacturer" argument might make sense.

But in reality where AI operates the models as well, the analogy is more like "Give me a rough idea and we'll use our pencils to write your letters for you!".

What a bullshit argument. Please don't post bait on HN.

Why it's bullshit: the pencil will not help you create an image. It's literally just a carrier for a medium, graphite. AI will create a high quality image in response to a textual prompt, even if the operator has trouble drawing a triangle. It's good enough to fool some people into thinking that images could be real photographs.

I'm sure you understand this just fine. It's very disrespectful to come in here wasting people's times with such specious arguments.

Might is irrelevant and doesn't prevent any abuse, nor does it foster innovative environment. Both the EU and the US, need to pick a side. Either they should make it illegal and establish precedent, or just let go.

>Either they should make it illegal and establish precedent

At that point we just wait for the next Chinese open source model.

This is also why we won't get a Big Tech crackdown any time soon of any significance; and would not have even under the previous administration.

Big Tech is the US' golden goose in the race against China. Deepseek shows China is at the doorstep, much closer and more capable than previously assumed. Any thoughts politically about how we can simultaneously crack down on Big Tech, while keeping China in check with sanctions, just went out the window.

Expecting companies to actually license copyrighted material instead of mass infringement is fairly reasonable given they've demanded the same for decades from the populace.

"China's not going to respect those laws" is kinda beside the point. If they suddenly decided to cut everyone in the nation's pay in half - or double it - that would have no bearing on what is right for you or I to do.

> If they suddenly decided to cut everyone in the nation's pay in half - or double it - that would have no bearing on what is right for you or I to do.

They literally did exactly that relative to the salaries of the rest of the world, and everyone took them up on it.

In retrospect, keeping China a weak communist nation was so easy. There was even internal dissent in the late 80s. It simply required refusing to make trade deals. US and worldwide wages would have been higher, discontent would have continued fermenting, the party would have remained relatively weak, human rights would not have been so easily sold out to the lowest bidder, the US would probably not have lost 6 million manufacturing jobs in a decade (3x the number of jobs in SV), and we blew it.

Capitalism doesn’t care about rights or how much you get paid — the only relevant metric is how much money you make so this was inevitable in capitalist America.

> In retrospect, keeping China a weak communist nation was so easy.

This is extremely naive, and fallacious.

As policymaker, you do not know beforehand how countries are going to develop over a 40 year period (not even your own country :P). Thus the only realistic option would've been a catch-all sanction regime against... possible future geopolitical rivals? Non-democratic nations? States with different cultural values? No matter which you pick, sacrificing trade like that would've been extremely expensive and limiting for US growth (might've included India, Africa, Vietnam, Thailand, Japan, Europe, Russia, depending on what criteria you pick).

You might have seen other countries jumping at the opportunity, filling the gap and benefitting immensely in the process, like the EU, or India, Russia, Japan, some pan-African Union... The only certainty in the outcome is that the US in such a scenario would NOT be as wealthy as it is today.

China isn't meaningfully communist. In general them stealing our jobs is probably a good thing on the whole. It's a complex issue, and I would be happy if we were pressuring China hard because of it's human rights abuses but global wealth inequality going down is something to be celebrated, protectionism sucks. I know it's complicated, but I can't be too mad about our wealth being "stolen" to lift people out of poverty, it's something I think we should be doing willingly.

Great sentiment, but let's start by doing it domestically first.

I think for my viewpoint to be internally consistent it needs to happen internationally as well as domestically. ofc this is all a pipedream anyway, there's barely an appetite for lifting the people in the tenderloin out of poverty and I'm trying to convince people that we'd all win by tending toward a "humanity vs the universe" viewpoint that's based on the idea that all the folks on the planet deserve to have it good.

"when you're used to privilege, the loss of it feels like oppression"

It will hurt to fix this, but I don't think it needs to hurt that much I think it would hurt a lot less if we were actually trying to make it happen rather than occasionally being dragged kicking and screaming in that direction.

(i do not have any meaningful ideas how to bring about this kind of change, (maybe a fake giant squid alien in manhattan? :P))

The thesis at the time was that through engagement and free trade we could gradually (over decades) transform China into something closer to a free-market multiparty liberal democracy. That policy obviously didn't work — in fact it has been a complete and utter failure — but even in retrospect it wasn't completely crazy or stupid. It could have at least partially worked if someone other than Xi Jinping had replaced Hu Jintao. Unfortunately a lot of major geopolitical trends come down to random luck and unpredictable individual personalities.

Now we have to pivot and focus on containment in Cold War II.

The economic liberalization of China and its participation in global markets has led to the fastest and most widespread reduction in poverty in human history. Even if we could have kept China in the dirt, it's extremely questionable if that would be the right thing to do.

The US government shouldn’t care about the right thing to do, only what’s good for the US people. It’s easy to say from a position of postwar supremacy that countries should be somewhat altruistic, but now the US pays the price, and will continue to do so.

You can't license 20T of tokens, I guess it's hard to grasp how big these datasets are.

"Can't" or "don't want to"?

OpenAI is talking about spending half a trillion US dollars, they have the money to license data.

In music, there is compulsory licensing and companies that use recorded music are able to make the economics work.

It needs to be repeated that these are not simply "tokens", they are the product of millions of individual people that are being appropriated for the financial gain of a very few other people.

Imagine if the US (public and private sector), transferred 2% of GDP to Warner Bros

>"Can't" or "don't want to"?

Can't. Even if someone has the money (I truly doubt), you can't contact millions of copyright owners (as you report).

If it's impossible to do it legally, then they shouldn't be able to do it. Violating one person's rights is illegal, but violating a billion's rights for profit is fine?

I'm in support of them being able to do it, but the right avenue is by working and lobbying hard to change antiquated copyright laws. Being able to disregard copyright only if you have enough billions of dollars on hand is the worst outcome. It's literally laws that only apply to the poor.

Be careful where you're going here. If you maximally/strictly interpret copyright law, the Internet Archive (including Wayback Machine) is largely violating copyright all the time. (WAY beyond the ongoing dispute with the publishers over the lending library.) Most web content is non-permissively licensed.

I don't believe Internet Archive should be permitted to disregard copyright wholly either.

Or because the results of these models are so transformative that you could pass it off as fair use.

Going with the flow here, does that mean if I build a little script that downloads just enough movies, songs and books from the internet I don't have to obey the current law, because it is a) too expensive, and/or b) impractical?

I'm sure you already see the folly of that argument.

Anyhow, flowing on, the allegedly totally inefficient governments of this world routinely contact millions and millions of legal entities, and many of them are poorer than Microsoft, Google, or even OpenAI, yet they somehow manage. So it seems to be practical.

Of course, that does not answer the cost thing, we all know governments just print more fiat money...

So we have been told that IP is indeed property and the property owner has a right to compensation for use. Nobody ever told me that I just have to be blatant enough to be scot-free. And I guess Sony, Warner Bros., Atlantic et. al. didn't get the memo either, or why would they sue a single university student for 4.5 million dollars? [1] This seemed and was much too much for a single university student to pay. So "too expensive" is off the table, too. Weird world.

[1] the Tenenbaum case. Tenenbaum was lucky but still broke afterwards.

>I don't have to obey the current law,

There is currently no law that states it is illegal to train a model on copyrighted work.

Well, the plaintiffs in the copyright suits are arguing that copyright law already does. But I can tell this conversation is not going to go far.

If that's the standard, then it is worth noting that we are talking about companies that are trying to do something that literally (as far as can be proven today) can't be done (build an AGI).

Contacting millions of people is something many businesses on earth do.

If these companies are already engaged in trying do do something that quite literally can't be done (again, as far as can be proven today), it's not out of line to ask them to at least try to do something that many other companies actually do in practice (pay lots of people).

It's important to be very clear that this is something that could be done, but that the AI companies do not want to even try to do.

It's not a problem for the music and video streamers. Get real. They could even have an AI do it for them!

Then you can't use it. Or maybe it's time to abolish copyright. Turnabout is fair play: if copyright binds me, it also binds you. If it doesn't bind you, it doesn't bind me. Anything else is pure corruption.

There are other things about the AI industry that don't foster an "innovative environment":

- companies being allowed to spin fairy tales about their products' capabilities

- no real consequences to enabling scammers, copyright thieves, and misinformation factories

That stuff might be good for securing short-term investment - and it befits a society obsessed with cryptocurrency and sports gambling. But it doesn't seem good for building meaningfully smarter computers, just dumber computer users.

Force them to open source it.

Force them to open source it!

Creation of new laws, and enforcement of existing laws usually trails behind the bleeding edge of tech. It's part of the risk tech companies accept when they're doing something new which might possibly be a bit dodgy.

This article seems like a pretty dishonest summary of a legal advisory that begins by saying:

> AI systems are at the forefront of the technology industry, and hold great potential to achieve scientific breakthroughs, boost economic growth, and benefit consumers. As home to the world’s leading technology companies and many of the most compelling recent developments in AI, California has a vested interest in the development and growth of AI tools. The AGO encourages the responsible use of AI in ways that are safe, ethical, and consistent with human dignity to help solve urgent challenges, increase efficiencies, and unlock access to information—consistent with state and federal law.

It's impossible to understand this as a statement that AI companies are a "legal clusterfuck" or "may be entirely based around criminal activity".

The actual warning seems to be not to do things with AI that are also illegal without AI. Don’t create illegal fake content, don’t make false claims regarding the accuracy of your AI and don’t produce AI systems which are discriminatory against a protected class.

Seriously. I did not realize Gizmodo was at that point with opportunistic clickbait, but apparently it is.

edit to expand: from the headline I thought this was going to be Bonta coming out against the argument that AI training is fair use, which really would at least arguably apply to "almost everything" the companies make. But no, he's just saying not to do things that AFAICT they already all ban in TOS.

If they continue they might get a strongly worded letter.

Fine of five hundred dollars and a personal donation to a re-election campaign required.

If the US doesn't do it, China will and this argument has nothing to do with deepseek although it does prove the point.

"China does it" is not an argument for allowing things which are unethical.

There's levels to that, it's not an argument that can be dismissed that easily.

High level CA politicians larp as national politicians way too much

Given the economic contribution and population of California it seems appropriate. See "map of states with a population less than LA county"

That people want to pack in populated, dense urban counties in tiny apartments doesn't mean anything.

It means that county is doing better than many states.

Except people and companies can change their resident state much easier than they can change their nationality. Nation-level policies do not necessarily work at state levels.

They are national politicians (as is every politician in the nation). I think you mean "Federal".

That aside, state governments are supposed to heavily augment federal laws with their own state laws and regulations.

California doesn't need or want input from a bunch of states with little to no AI (or even IT in general) industry footprint about how those companies should operate.

People fundamentally misunderstand how "AI's" work. They merely regurgitate statistically what is fed to them, and that includes vast corpus of data, and they are a black box. Hence of course they will disseminate false and misleading information.

They don't actually have "intelligence" or agency. There's also not much we can do about the hallucination problem.

So much like people then.

> here's also not much we can do about the hallucination problem.

Call them what they are: errors.

"These tools suck; QED no one should be held accountable."

I must have misunderstood you.

The code of federal regulations has over 185,000 pages and no one has time to read it all and anything anyone does ever may thus be illegal.

They're usually not secret. You want to sell a product in California and lie in your advertising - you should probably check the laws about false advertising in California. The bigger you are, the more you can afford to hire a lawyer to know that for you.

the original title uses "California", not "Cali".

“Cali”? No.

What? It is.

Cali is a big city in Colombia.

Only LL Cool J can call California "Cali" without derision.

Ok, deride away. You may find it useful to know that "Cali" is a frequently used shortening of "California". Good luck with the crusade.

Oh I'm very familiar. It's still ambiguous and unnecessary in this context.

I mean yeah, but perhaps going after the provable and wholesale abuse of copyright laws might be a more fruitful approach.

https://www.reuters.com/technology/meta-used-copyrighted-boo...

After all its actually documented. Its also not likely to be fair use.

The proving now harm bit is going to be difficult, and bad on the old trumpian optics.

copyright at least forces some of his billionaires to fight with other billionares to come to a conclusion as to "what is good for America"

Google making a database of all web content and serving up snippets of that content in search results was ruled to be fair use. So it seems plausible that doing essentially the same thing but with an AI rather than a traditional database could be ruled to be fair use.

Its hardly a sure thing but their position does seem to be at least somewhat supported by precedent.

https://www.supremecourt.gov/opinions/22pdf/21-869_87ad.pdf is the most recent SCOTUS precedent covering the definition of "transformative" for the purposes of fair use.

The tl;dr is that you have to look at the impact on the market for the use to figure out if it's transformative [1]... which means it's extremely unlikely that training for AI is going to be considered "transformative", and thus they lose the first factor. Given that AI is absolutely reamed on the fourth factor (especially now that they're paying people to use their content for training, that's basically a concession on the fourth factor), there isn't really any grounds for them to claim fair use.

[1] Yes, it's pulling the fourth factor into the first factor, it is a rather garbage opinion, but it is precedent as of 3 years ago.

Unfortunately, nowadays what the laws say and what's legal are two different things

If you want to be pedantic, fair use isn't about what the law says, since it's entirely derived from judicial precedent rather than statutory text. It wasn't until the 1976 copyright reform that fair use was incorporated in statutory text, but as anyone who's followed any cases on fair use should know, there's never argument based on the statutory text itself.

Fair point, though it doesn't seem like the courts are putting much weight on precedent for their decisions either

Now compare this to the legal atmosphere that DeepSeek was created in.

Oh, the "we'll give you money under the table so you can appear cheap as hell, but you'd better give us all your data on the imperialists, Comrade" regime?

Wow, now that's what I call lawfare! Vol. 23

Can Cali's AG copy this letter to DeepSeek, lol?

This smells like a proxy maneuver against Big Tech for their posturing toward the Trump Administration.

If the government cannot give you a definitive opinion whether you are breaking the law, the government has no mandate to rule.

Rule of law means having clear guidance on what is or isn't illegal. Vague guidance that everyone in the industry "might" be breaking the law isn't responsible, it's setting up a mechanism to trade favors and selectively prosecute enemies to reward friends.

Be better, guys. This isn't the right take.

Rule of law means that an unbiased judge with an understanding of the relevant precedents allows two litigants to argue their cases on the merits of the case and of the evidence, and then when the judge or jury renders a judgement, the society accepts the judgement and allows any consequences to be carried out without regard for who the judgement is for or against. It has nothing to do with "clear guidance" because what is clear as day to one person may be totally unfathomable to another. I guarantee that you or I could find two statutes together that make perfect sense to me but which you do not understand, and vice-versa. That's why we have attorneys and judges.

People have a right to know, in advance of an action, whether the AG or DOJ will wish to prosecute it. If the AG or DOJ haven't made their position clear, you should be able to ask, and they will give a clear answer.

That is an extremely basic rule of civilized governance. You can't wordplay around it.

This is not the entire legal system, but it is a critical part of it.

This is pretty inaccurate. Absent actual laws that people are aware of, you described rule of man, the opposing governance form.

Rule of laws means there are known laws everyone has to follow no matter who they are.

https://en.m.wikipedia.org/wiki/Rule_of_law

No, that's not how it's supposed to work. Laws are supposed to comply with the vagueness doctrine.

https://www.law.cornell.edu/wex/vagueness_doctrine

That's not at all how the judicial system works. Do you know what legal precedent means? Many things are uncertain legality until tested in a court of law.

No, whether the AG is has the correct interpretation of the law is tested in court.

The AG's stance on whether an action is a crime, and their policy towards prosecution, does not need to be tested in court. That is something they can communicate without litigation.

That sounds like what they're doing. This whole thing is about the AG writing two legal advisories, not explicitly beginning the litigation process.

there seems to be an error in the headline – Cali is a city in Colombia, but the article is about California?

HN's title character limit is 80 (and the title is exactly 80 characters), so it was abbreviated to fit.

CA would have made more sense as its an actual acronym.

That's a bit US-centric.

CA isn't an acronym, it's a postal abbreviation.

Internationally, it conflicts with Canada.

It was abbreviated incorrectly and it is ambiguous or even misleading in its current form.

> Otherwise please use the original title, unless it is misleading..

I would vote for:

"California AG to AI Corps: Practically Everything You’re Doing Might Be Illegal"

the cat is of the bag, there is no point in trying to fight it at this point if data is accessible. models will be built on top of models and will become cheaper to train. in fact, u can fight these battles now only bc they are big companies and training is expensive. if training/inference improves and becomes more at the edge in the hands of users, nobody will stop anyone else from consuming whatever data they want (raw data, other models, whatever) into their own models just like we consume whatever data we hav access to and build our own mental models. all you can do is simply not share your data.

Thomas Sowell definitively disproved the idea that disparate outcomes are ipso facto proof of discrimination. It's past time this line of jurisprudence was put to bed.

I assume you are referring to this section:

> Create or sell an AI system or product that has “an adverse or disproportionate impact on members of a protected class, or create, reinforce, or perpetuate discrimination or segregation of members of a protected class.“

It's a straw man to characterize that as saying disparate outcomes ALONE are why AI systems might be discriminatory. They may well can be, and likely are, actually embedding biases in the models.

The simple examples of how many systems often prefer "he" for doctors and "her" for nurses shows that bias in datasets results in bias in the models. Yes, that is a result of the dataset and reflects the dataset (and maybe even real world statistics on doctors and nurses!) but it does mean the system may treat women and men differently when there is no legal justification to do so.

I reject the notion that legal justification is necessary. Your LLM prefers "he" for doctors and "her" for nurses, and therefore you can be prosecuted by the state? That's ridiculous.

All such Harrison Bergeron-style speech-policing is antithetical to a free society.

No, I'm not saying the he/she thing is worthy of prosecution. Straw man #2, nice. It is just an undeniable illustration that bias is encoded in machine learning models.

Of course it requires some actual harm before the law is involved. Perhaps AI recruiting systems favour resumes that match the "correct" gender to profession. Perhaps a hypothetical AI sentencing recommender gives stiffer penalties to people that live in high crime areas; after all, statistically they are more likely to re-offend - that's just not an acceptable sentencing factor. Perhaps a house appraisal AI accidentally recreates redlining.

These are just hard to prove, so my example is there to demonstrate that even though it's hard to demonstrate we should take the risk seriously.

>or create, reinforce, or perpetuate discrimination

Perpetuating "harmful stereotypes" reasonably falls under this language. Trying to correct the bias of the dataset, or worse, correcting the bias of reality, is a fool's errand. The same motive produced the Google AI images controversy last year and constitutes a substantial percentage of instruction fine-tuning for "alignment".

I agree that care should be exercised in rolling out algorithms for sentencing guidelines.

The last thing we need is for California to keep adding more regulations, as CA’s incredibly bloated government has for a couple decades, and irresponsibly killing American prospects in the race for technology superiority. Rob Bonta is already a bad AG for his repeated blatant violation of constitutional rights. If this headline is truly his stance, he needs to go, and immediately. We have no time as a nation to tolerate people like him at this critical time.

Reply on news.ycombinator.com