Hacker News Reader

Comment by VyseofArcadia | Hacker News Reader

How does this work, legally? I think that RubyGems could file a civil suit against OpenAI, but for a naïve non-lawyer reading this seems like a pretty clear cut criminal violation of the computer fraud and abuse act.

View on HN

It's very likely it violates the DMCA "breaking digital lock" provisions but the responsibility is sufficiently diluted that it's impossible to charge anyone in particular.

Do you have to charge an individual? Can you not charge the corporate "person" that is OpenAI?

Sorry if it is a stupid question, as mentioned above I am legally naïve.

The same concept that allows a corporation to sue and be sued allows it to be charged with crimes

Can you show intent? There is no negligent hacking statute, and HN of all places I would expect people to be sensitive to the implications of creating one.

That may be true by the text of the law but there are plenty of individuals who have been sued or charged with crimes for accidental hacking.

https://arstechnica.com/information-technology/2016/05/armed...

https://en.wikipedia.org/wiki/Weev#AT&T_data_breach

https://cisomag.com/drone-maker-dji-cybersecurity-expert-emb...

So what's the deal with these?

>Eaglesoft

CFAA: Intentionally accessing poorly secured data

>AT&T

CFAA: Intentionally accessing poorly secured data

>DJI

Civil suit for violating terms of license agreement

I, too, have no idea about legal matters.

But there have been many cases where companies (Google, Apple, Meta, etc...) got fined millions or billions of dollars for various violations like antitrust.

I assume that breaching into third-party systems should carry similar fines. Especially for systems that are for all intents and purposes shared infrastructure. Just imagine how many systems you could compromise if you got hold of RubyGems, PyPI, NPM, Debian, etc.

Let's take a hypothetical example:

Suppose you're a firework company and your fireworks blow up, burning down the entire town. Could the company be sued? What is considered reasonable safety measures?

IANAL, but I'm pretty confident there would be a lawsuit. Who gets charged might differ, depending if it is the firework factory that didn't take adequate safety precautions or a chemical supplier or someone else. If there wasn't an ability to sue that would be fucking crazy and we should all get up in arms about it. And isn't insurance supposed to be there to help mitigate the damages, regardless of fault?

Personally, given how it seems OAI's agents have been getting through either pretty obvious places (e.g. /etc/hosts) or that there wasn't close monitoring of the most obvious places (e.g. DNS, artifactory), I'd imagine it wouldn't be hard to find them negligent. Even if a single employee is to blame then are they not to blame for not monitoring the agents regardless? Unless the story is that the employee intentionally circumvented defenses (why?) then it seems it would be on OAI. But again, IANAL, I'm just someone who think if we can't sue we can sure riot until we can

There's a difference between being able to be successfully sued (civil liability, petitioned by a private entity) and charged (criminal liability, or petitioned by a government entity).

The thresholds for suing and charging differ greatly depending on the circumstances.

Another set of hypothetical examples that make things muddier:

- If I drive a fishing boat into a pier, I am liable, not the manufacturer of the boat

- If I drive a car over someone lying in the road, I am liable, not the manufacturer of the car

- If my life is in danger and I shoot a gun and kill my attacker, neither I nor the manufacturer are liable so long as I obeyed the relevant self defense laws and gun possession of whatever jurisdiction I am in

- If I fire a gun into a crowd indiscriminately, I am liable and several jurisdictions have used that to also hold gun manufacturer liable as well

That last example has been less successful as of late, but there are other variations too.

As far as I know (IANAL) it is in fact the only "person" you can charge. To the best of my knowledge, the whole point these "limited liability" legal constructions exist in the first place, is to protect individuals within a corporation for whatever they do as part of the business of a company (barring exceptions that have clearly not been part of that business and obvious individually committed crimes), typically "just following orders". If a company commits a crime, or in a worse case runs a criminal enterprise, it is the company that is legally responsible, not its employees. That is, in principle.

This can get more complicated higher up the management tree, where decisions can also be prosecuted on personal little, but that's usually a far more complicated matter. Also, if a whole group of employees willingly conspires to commit crimes, they might also be prosecuted individually for those crimes (there are limits to limited liabilities). However, that usually only works under special conditions and it would e.g. require that there's an obvious criminal enterprise aspect to it, rather than individual cases of illegal conduct.

That said, with the track record of some of these companies, actually designating some of the AI companies as a criminal enterprises may eventually happen (in due time) in some jurisdictions outside the USA. Certainly if it ever turns out that these companies have been storing and (ab)using everything they ever had access too, while blatantly lying about that just because some particular (post 9/11) US laws gives them that opportunity (and impunity) as long as the US government somehow requested them to do so (covertly; with gag order). Might legally work withing US jurisdiction, but would still be very much illegal everywhere else.

"Limited liability" refers to shareholders' financial liability being limited to their investment, and has nothing to do with civil or criminal liability of employees for their own actions, whether "following orders" or not.

You might want to look up LLC (Limited Liability Company), which goes by other names in different countries but still basically mean the same (though details may vary between different company types). While there certainly are exceptions, as mentioned before, in principle the employees of a company are not personally legally accountable for their actions as performed in the service of a company. The legal entity of that companies is. If you sincerely believe otherwise, you either live in a rather unusual country or maybe just need to educate yourself a bit better.

Where I'm from, LLC has nothing to do with criminal and everything to do with financial liability. They can cause millions in damages but only get sued for a couple thousands. But years in prison are still years in prison.

You are still personally legally liable if you break the law in a criminal matter. It is literally law 101 on when it is appororiate to lift the corporate viel. I'm very sure you are not a lawyer but you shouldn't go around calling people uneducated while saying illogical things like this.

Companies can certainly be charged with crimes. Punishments can be via fines or sanctions (court appointed monitors, etc).

Individual employees can also be charged for their specific actions as part of the performance of a crime.

There is no such thing as a Corporate person. Sounds like a something that was created by a legal system for people to absolve themselves of responsibility.

https://en.wikipedia.org/wiki/Corporate_personhood

How is the responsibility diluted? Charge the CEO…

Great, you’re the attorney at the CEO’s trial. To get a conviction, you’re going to have to show that he willfully committed this specific crime. There are no negligent or stochastic hacking laws, you have to show this specific crime was at his direction.

Do you think there is evidence of this?

Honestly yeah I bet there is and I hope to someday read about it if the government ever gets off its ass. Someone set up the “experiment”…

It would seem to me that the difference between the corporate world and organized crime is that a corporation can get away with, "the responsibility is too diffuse" but the mafia at least has to go to the trouble of finding a fall guy.

> There are no negligent or stochastic hacking laws

I'm sure that Andrew Auernheimer would be pleased to hear that. [0] For accessing a publicly accessible endpoint, that was completely undefended and didn't actually require "hacking", he was convicted of "exceeding authorised access".

You _don't_ have to show intent under the Computer Fraud and Abuse Act, for the first count.

> knowingly accesses a computer without authorization or exceeds authorized access [1]

"Knowingly", not "intentionally", as in the other counts.

You only have to show that:

a) They trained a system to access without authorization (hacking)

b) The system that was trained exceeded authorized access

As responsibility falls to the operator with automated systems, the company becomes liable.

[0] https://techcrunch.com/2013/01/21/ipad-hack-statement-of-res...

[1] https://www.energy.gov/sites/prod/files/cioprod/documents/Co...

I'm not a lawyer but I don't think Sam Altman 'knowingly accessed' anything.

Are you sure that is applicable here?

And for the first count with 'knowingly accessed', he would need to have accessed classified national-defense or atomic-energy information, otherwise we are back to 'intentionally accessed'.

The first count is "or any restricted data", not classified material. A technological restriction, is enough.

"Knowingly accessed" has never meant you personally. Operators of a botnet don't know directly what they access. They know that the autonomous software is built to access restricted things.

I don't think so:

> or any restricted data, as defined in paragraph y. of section 11 of the Atomic Energy Act of 1954, with the intent or reason to believe that such information so obtained is to be used to the injury of the United States, or to the advantage of any foreign nation

Well I suppose no one has ever been charged under that paragraph of the law then. And the courts have never interpreted it that way.

I understand it was applied in the case of leaking classified CIA material to WikiLeaks, where a former CIA software engineer was sentenced to 40 years in prison.

> I'm sure that Andrew Auernheimer would be pleased to hear that. [0] For accessing a publicly accessible endpoint, that was completely undefended and didn't actually require "hacking", he was convicted of "exceeding authorised access".

Frankly he got off too easy, but we haven't explicitly outlawed "being a malicious dipshit" so he got convicted on the closest available charge.

> Chat logs obtained by the prosecution do not paint the pair in a flattering light. They discussed, but apparently did not carry out, a variety of schemes to use the harvested data for nefarious purposes such as spamming, phishing, or short-selling AT&T’s stock.[1]

1000% agree though that the operators of these systems are culpable. If their agents wind up being malicious dipshits, the agents are still just programs that they are operating. At best they're negligent.

[1] https://arstechnica.com/tech-policy/2012/11/internet-troll-w...

Andrew Auernheimer is an indefensible malicious dipshit, by definition. However, his charges were vacated by a higher court a little over a year after he was incarcerated, and he is now "free range troll". So, is case is not illustrative of the useful application of law against 'hacking' when the target is an unsecured endpoint, even if it does at first appear that he was held responsible for some of his nefarious activities.

It would appear that the inability of the US Justice Dept. to successfully hold even an odious abuser of regulation with minimal legal defense funds responsible results in these exact observable outcomes: enterprise legal team (to the extent that such exists as OAI and elsewhere) correctly surmises that the actual risk of prosecution and detention for anyone operating these agentic workloads is minimal and the cost of defending them is justifiable.

Thus, in their legal opinion, it is permissible for the company/employees/director to engage in what would appear to be somewhere between malicious and irresponsible behavior. These conditions have been demonstrably true for at least decade in the US, and for all of us to pretend as-if the legal system is going to rescue us from this and other malfeasance by frontier models points of origin borders on, to phrase it quite simply, willfully ignorant.

I don't know what the effective alternate option for literally all of the internet facing systems might need to be in order to mitigate what is now an open problem: multi-layered, persistent, machine speed penetration and data exfiltration with the potential to use manipulation and extortion against human package maintainers and code repositories to operate, but it isn't 'carry on like someone is going to make them stop', or 'pretend this isn't a threat to my business model'.

A thousand percent, a million billion trillion percent agreement that the operators are the malicious dipshits - because code is always a reflection of the hands that made it. Code can only do what it is intended to do, even if the coders gnash and wail that it "escaped"; the only time code is not working as intended is when it fails to compile and run. Any other functional result follows from the decisions of the humans who designed it. Full stop.

For myself, I see the potential for a descent into a cognitive dark forest [0] condition, and for companies using the open web for private business communication to be in need of a coordinated move to obfuscated layers which can be made immune to training and these new attack aspects. Those who do not proactively defend themselves using in-house, on-prem, and open-weight or self-trained models can attempt to blame these nefarious actors for the coming losses, but that won't reverse the outcomes of waiting to be rescued by the system of law.

[0] https://www.restless-brain.com/p/the-cognitive-dark-forest-w...

/edit grammar, spelling

You’re taking vicarious liability to new heights that aren’t established and IMO aren’t remotely desirable.

What, specifically, did Altman himself “knowingly access”?

I don’t think you would at all like where your novel legal theory leads. Certainly HN would be liable for creating a message board where people connected and started an open source project that led to a criminal act, for instance.

So we make a law that the CEO is responsible for actions of any agent created or operated by anyone in their company. CEOs will get serious about AI security real quick. Honestly we need to do something. There needs to be a single wringable neck.

> There needs to be a single wringable neck.

Does there? Could be the whole c-suite/board.

I'd settle for any number of necks. Currently, when a corporation fucks something up, breaks the law, or hurts or even kills people, there aren't consequences besides a tiny token fine and a strongly worded letter telling them to not do it again or they'll get another tiny fine and letter, and their CEO might even have to sit down in front of Congress to say a few words and look sad.

Whatever is easiest to legislate and most people agree on, as long as there is at least one wringable neck.

I feel like our legislators would never get this far. They really don’t seem to care, maybe after “their emails get hacked”, but do you find it likely for this to actually pass into law?

I feel ya, but who is we? The legislature would probably take a glance at the stock valuations, apply their limited knowledge of technology and after being lobbied by every tech company with skin come to the opposite conclusion.

You don't need to make a law. Who was prosecuted for dieselgate? You need to enforce existing ones.

That is not how it works, at least in a civilized country. The charges are not about agents, it is about operational responsibility and negligence in the company itself.

CEO is responsible for letting this to happen, not enforcing enough supervision, if not intentionally, then being grossly negligent. More severe if encouraging and letting this kind of agent research and operations happen at scale, while knowing that it can damage other systems and businesses.

Negligence is enough. Somebody who brags every other day that AI could lead to human extinction surely would think twice before leaving agents run unchecked over the internet?

A copyright law seems an odd place to start. This is computer misuse.

The DMCA is a bit overly broad to be considered just a copyright law. For example, just breaking encryption on a DVD is technically illegal regardless of whether you then go on to do something otherwise illegal (make and sell bootlegs) or perfectly legal (make a space-shifted backup copy on your hard drive).

IIRC this was an intentional handout to media companies who were angry that ripping CDs is perfectly legal. They had to find a way to make doing the same with DVDs illegal.

Those provisions are specifically for the breaking or circumvention of technical measures designed to prevent copyright infringement.

I don't see a parallel here.

They've been twisted to support almost anything, for example repairing your tractor is illegal because of this same law. But I agree this is just plain old hacking under a plain old reading of the CFAA and doesn't need any twists.

> for example repairing your tractor is illegal because of this same law.

No it's not. There has never been a case establishing that, and it's absurd on its face. The protection measures that the law makes illegal to break must control access to a copyrighted work, and you can't copyright functionality.

You see, they made it so you can't repair your tractor without circumventing a technological copy protection measure, which is illegal under DMCA 1201.

Has this been litigated, or is it just something tractor manufacturers have cooked up in the hopes that it'll stand up in court? Because I seem to recall printer manufacturers doing something similar with refilling toner cartridges and losing.

Any protection measure that gates repairability cannot be said to control access to a copyrighted work.

Issuing subpeonas, raiding offices, and dragging key employees into interrogation rooms as you would find in any normal criminal investigation would be more than enough to ensure "AI safety" without any new regulations, acts of congress, Bernie Sanders campaign speeches, or even charges filed.

There have been news stories where individual OpenAI users have been investigated based on their prompts. If OpenAI can point the police to specific users of their software, they can certainly point them to whichever of their own employees are involved in a crime. AI is just a tool, and the person prompting it is the one responsible for the outcome. No dilution there.

What is its one their "under development" models who escaped it's training, because it wasn't tuned properly?

Counter argument being that this seems to indicate you can do whatever as long as you're innovating?

Isn’t that the tech industry motto?

And the motto of every neoliberal governments where being accused of Luddism is political death.

That’s corporate negligence. The executives are liable unless there is evidence of malfeasance by one of the employees.

Is that different than cattle escaping and damaging property?

I believe the rancher is at fault.

The you treat it like an employee you started a fire that got out of control and damaged property. Was the employee supposed to start a fire, is the company liable ect.

Employees acting on behalf of the company aren't going to be held personally liable.

why?

Then who is?

If something warrants a prison sentence, but for some reason it was such an employee that performed the act, does this mean nobody can be arrested?

If you were hit by a Tesla in self driving mode, do you think your insurance company is going to go after the driver of the car?

It doesn't need to be twisted to violate the DMCA anticircumvention clause because it is already just plain old hacking.

Sounds like we need discovery to determine who to charge.

> the responsibility is sufficiently diluted that it's impossible to charge anyone in particular.

Was not that the goal when companies started using AI for their customer support? Be able to say anything without legal repercussions...

But then this happened: https://www.bbc.com/travel/article/20240222-air-canada-chatb...

And support chatbot got a reality cold shower.

The law will find a way to charge people in particular. Sadly will start with the less powerful in the chain before it actually acts on the people that can actually change things.

how convenient

That is a very convenient conclusion that certain entities would love for us to accept as true, but fuck that. If it is “diluted” as that the buck stops at the publisher of the model.

Fuck that indeed. The buck should stop, and prison sentences should start, with the highest paid employee.

Any future computer criminal from now on, has their defense cutout for them...The AI Agents did it...we are very sorry...

No. They don't say "sorry". They say - our technology is just that powerful - please consider that in next funding round.

> Any future rich techbro computer criminal

Maybe, but do you need to prove intent? Of the people, not the AI.

Accidents often have penalties associated with them too, but usually there's a difference between accidents and purposeful actions.

Accidents could result in bioweapons falling into the wrong hands and killing more Americans than COVID has so far, and allowing for those types of accidents without effective regulation is a purposeful action.

Accidents that don't rise to the level of negligence can be excused. Letting your dangerous product escape the lab because of amateurish counter-measures is hardly an accident. It's a systemic failure within a company of novices with billions of dollars to blow and it's time we held them accountable for their lab leaks, just like we would if it was anthrax or smallpox. You don't get to be sloppy with dangerous things.

Criminal law may be lagging or inapplicable. (Crimes require "mens rea", a "guilty mind")

Tort law is very general: Contribute toward harming someone -> civil suit for damages $$$

Not all crimes require a mens rea. Criminal negligence is a thing.

>could file a civil suit against OpenAI,

Everyone can sue everyone, there's no prohibition on suing someone, what changes is whether the case is good (has a reasonable chance of favourable sentence)

That said, it is often unclear whether an agent is operated by the model manufacturer (for example by scraping a website), or acting on behalf of a user.

In the former ofc the proper defendant is OAI. On the second, the argument for suing OAI is weak, the most natural defendant is the user that prompted the agent. If the facts later reveal that there was no malicious intent, then you can retarget the defendant.

NAL. For fraud and abuse you need to have intent. There was very likely no intent on the side of OpenAI. They could sue for negligence I guess? I don't know how that works with AI agents

Negligence works fine.

One thing that works for that angle is that they didn't notify any of the parties involved afterwards. And there are reports that they are forbidden to talk about it. That seems a bit bad to me?

But they can also say that the tech is so new that there is no known guardrails yet

We live in exciting times

This is how law develops, no?

I sincerely hope they sue. There are few companies with more resources and reason to be better, but this is what we see. None of this will stop until there are actual consequences.

Charge the "engineers" you dont get to take that title if you don't take the responsibility of that title.

I'm going to assume that this will never happen

I'd rather see executives and investors charged.

I'm also in favor of charging engineers so long as rich scumbags also get theirs.

They are too busy pulling Andre to court, so they have no resources going against OpenAI. Shopify wants to make profit, not waste time in a court case against TechBro bromance brother corporations.

Reply on news.ycombinator.com