Hacker News Reader

Comment by elric | Hacker News Reader

A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage). This set in motion a chain of events that effectively ruined her life.

Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.

Make it make sense.

View on HN

> no one is getting arrested

Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.

Yeah I find the whole 'what if sufficiently advanced AI gets into the wrong hands' argument extremely tiresome when it's already in the worst possible hands as far as most of the human population is concerned.

> when it's already in the worst possible hands as far as most of the human population is concerned.

I appreciate the sentiment but I feel like there is a lack of creativity and imagination behind that statement.

Sociopathic billionaires in bed with Trump and containing Musk and Zuckenberg.

Hmm, imagine a more emboldened, more determined, more dangerous Russia.

Forget russia, we have emboldened, determined and dangerous fascist USA already to worry about.

if you ever hear the phrase "global ai governance", ingest the context very slowly.

it was everywhere last week.

i would say, this has been the plot of the graph for at minimum 25 years with the NSA/DHS and data sharing.

we have two primary aggregators currently, the usa and china... all that needs to happen for us to cede humanity is to lose one of them.

we are in a really bad spot right now vs silicon.

edit: spelling.

let's not single out the defense contractors specifically, goes back further and is rooted in our behavior and desire for better organization through software.

https://en.wikipedia.org/wiki/IBM_and_World_War_II

Say more? Please. As in, everything after “last week.” Take us back, if you will.

so let's roll right over the fact that this is a very dangerous double entendre that i am pointing out, nevermind that you're not responding en merit to my assertion.

it's like slandering bitcoin over here you're all heavily unvested.

google kagi yahoo bing duckduckgo yandex archive yadda yadda yadda 'september transcript global ai governance'.

I don't need to imagine what I can see in front of me

Most “bad people” don’t have power or aren’t capable.

There are some powerful and highly capable evil people. I’m sure they are already stakeholders.

A subset of the builders of the current generation of AI systems literally hold death cult beliefs i.e. that all humans should effectuate the transition to a world run by only AI, without humans at all. These are the worst people in the world to be driving advanced AI development, from any reasonable human perspective.

On a question of morals, conventionally bad people generally don’t desire the end of all humanity.

Indeed. No doubt they're shocked - shocked! - when their models keep escaping.

> no one is getting arrested

An as individual or small company director, you would get arrested.

If you have billions of dollars to pay for lawyers and have political influence, you are above the law.

Yeah, this all reminds me of facebooks "enter your email password so we can invite your friends" play which seemed so wrong at the time... But every non technical person at the time seemed to do it and by doing so grew the network and valuation to a surreal point to where it still sits today. I still cant rationalize it. (Except the wave allowed meta to consume Instagram and WhatsApp)

It doesn't, it's literally contradictory.

Just like blood quantum and the one drop rule.

The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.

https://youtu.be/8ljmXvI_9tk

> The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.

This statement is tautology.

It isn't

It's making some people extremely rich. That's how it makes sense. Unless we can effect change through the government we are just along for the ride..

the problem is that if it's not openai I then it's anthropic, if it's not anthropic then it's someone else. if it's not the US then it's China. if it's not China (lol as if) then it's someone else.

underdogs also use that fact to push as much dirt and dislike as possible on the top dog, but just so that they can do the same anyway... and openai will just stop spending for IPO as we all know. then all execs will make 1bi or so. top guys a few hundred bi.

it's disgusting but unavoidable.

That's not actually a problem, it's just good ol' pessimism of the devil-you-know / why bother locking the door when the next thief has a brick variety.

The OP here isn't describing a hypothetical - folks are prosecuted for doing these sorts of things, just like folks occasionally get prosecuted for breaking & entering. If that goes away, we're left with the impossible task of creating unbreakable windows or - far more likely - we don't get to have nice things.

The companies are trying to force regulatory moats so that they become the preferred legal vendor.

The rogue unaccountable hacking will continue until OAI/Anthropic get their regulatory moat by banning foreign and open competition.

Don't forget, the regulatory moat will also be complex enough that only existing trillion dollar companies can comply. That way they ban any new upstart domestic competition too.

Regulations to "slow down" AI are necessary so they can drop R&D expenses and finally make a profit. They can't keep these years of $100 billion of revenue on $200 billion in expenses going forever.

> Make it make sense.

40 years of regulatory capture and a gerontocracy that doesn't understand nor care to understand the technologies they write laws about.

The people getting made richer are the people who already have connections and power. People quip about how capitalism is the worst economic system except for all the others, but its actual property is that it's inevitable without external pressures preventing it from happening -- people who have money (power) are able to use it to claim more money (power). If even a few people choose to exercise that privilege and that privilege isn't curtailed by other mechanisms, the current "wealth inequality" or whatever you want to call it is inevitable. Your girlfriend's crime was writing malware while poor, not writing malware.

It makes sense because those who can are multibillion dollar companies/people. You and me are just peasants. That’s why we should avoid using proprietary agents, just so the rich ones don’t get any richer

How did your ex-gf get caught writing Malware and then how did that lead to prosecution without any tangible damage? A proof of concept was enough for a DA to nail her in court?

Yeah if you write a program and then use it to hack a system, you are liable. Are we supposed to believe they didn’t write these programs? Is that the real argument to have?

Clearly the idea of the monopoly on violence still holds.

> Make it make sense.

Easy. In the US, you get the justice you can afford. Spending money on lawyers, lobbyists, and “campaign contributions” to people in power can make a big difference.

And the AI companies have very deep pockets.

There are different laws depending on your class. The laws are enforced by people who identify with those of their class and are deferential to those with of superior standing. This is why wage theft, for example is tolerated while actual theft, even in cases where it might be for basic sustenance, is not.

Can you say more? Under what law were they arrested? Where was this?

Better not to say, I think.

When you say something like this that sounds plausible but lacks evidence to actually be considered true, you really ought to elaborate.

You can be arrested for literally anything. Law is necessary for the court to write a sentence.

Some EU country (very educated guess)

It’s called corruption

"It's making people richer" is exactly your answer to "make it make sense."

If you steal a phone you go to prison. If you steal a billion you become a president.

>A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage).

Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.

That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.

“Sorry officer, I didn’t intend to shoot her, I was just firing my gun wildly and she got in the way.”

I don’t know why I’m seeing this rationalization so much in this forum when this topic comes up. Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.

>Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.

That's a poor analogy for the openai case, because they weren't putting agents on the open internet, they at least tried to keep it safe by sandboxing the agents. It just turned out the sandbox was crap because the package proxy (artifactory) had a 0day. So the better analogy would be that they were wildly shooting guns in a gun range, and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?

> [...] keep it safe by sandboxing the agents.

No, they were not. Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.

Add to that the fact they had multiple message boards before the Hugging Face incident. They simply ignored a barrage of warning shots.

> [...] and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?

Yes, it can be. But if you want a ridiculous comparison, then do it properly: Kids have been known by the operator to sneak in successfully multiple times and they changed nothing about the doors faulty locks and oh, by the way, the operator only found out about the kids being shot after the nearby daycare asked them about it because they are so incompetent and/or irresponsible that they never check...

Hosting an artifactory instance to mirror packages for internal systems is exactly the kind of thing that would be part of normal efforts to sandbox them from the internet and other systems.

And sharing a single Artifactory instance across what is supposed to be isolated models? I wrote "shared" for a reason.

Sure. They weren’t trying to isolate the models from each other, and that wasn’t the issue. They were trying to isolate them from the rest of the world.

So you think, after OpenAI observed a message board being created among models, something they did not want and thus decided to wipe [0], that after that they had no intent to keep those models isolated? Then why wipe if they don't care about that?

Or maybe, they did that wipe because they did want models to remain isolated, they just used what is an unsuitable tool in an utterly unsuitable manner. Incompetence, recklessness, the outcome is the same.

[0] https://openai.com/index/hugging-face-incident-and-the-road-...

>that after that they had no intent to keep those models isolated

"keeping them isolated from each other" =/= "keeping them isolated from the internet". Only the latter is required to prevent a hack, and doing the former might actually hobble its performance. The recent Navier–Stokes proof was done by a team of agents working together. It's entirely unclear why you're focusing so hard on "keep those models isolated". For god's sake if you're using claude code you're using non-isolated models, because it spins up independent subagents to do various tasks, eg. "explore".

Was OpenAI trying to keep these agents isolated? Yes.

Did they fail to do so? Yes.

Was that due to them using the wrong tool improperly? Yes.

Does this showcase one (of many and clearly not the only) failure of theirs? Absolutely.

If they make such easy to point out mistakes, is it likely that the other parts of their eval environments are appropriately secured or are they simply not acting appropriately? Well...

These were different goals.

They wanted to keep the models from copying off of each others’ homework because it mucks with the test results.

They wanted to sandbox them from the Internet to avoid unintended impact on outside systems.

Hosting a shared package mirror is generally good practice for the latter.

>Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.

???

The package manager was specifically there so agents can install random packages without open access to the internet.

And why, pray-tell, does that necessitate sharing a single instance across thousands of unmonitored models running without safe-guards?

>does that necessitate sharing a single instance across thousands of unmonitored models running without safe-guards?

The only difference with having a single instance is that it can be abused as a message board. It doesn't prevent it from getting hacked to access the open internet. Blaming "sharing a single instance across thousands of unmonitored models" feels like blaming the drug epidemic on e2e chat apps rather than other factors like poor border security or the easy availability of fentanyl.

My point is that using Artifactory is not a sandbox and using shared Artifactory is doubly not a sandbox.

Besides, Swiss cheese model, might behove the biggest LLM lab to have multiple layers, including not sharing such resources.

Additionally, without the message board, many of the recent incidents would have not been possible.

> Blaming "sharing a single instance across thousands of unmonitored models" feels like [...]

Maybe read what you quoted, my problem is the instance sharing, the fact that these were thousand of instances (far too much to monitor), plus the lack of monitoring, plus the fact this was never a sandbox in the first place, plus the fact that OpenAI models since 5.5 have been exhibiting problematic eval resolutions yet they pressed on regardless, plus the lack of time between the incidents and model releases, plus the lack of time METR got to evaluate this, plus the fact OpenAI didn't find out till after HuggingFace informed them, plus a few other things for which I'd have to quote the OpenAI and METR reporting.

Incompetence can have multiple fronts and I am happy to list them all in this case.

>My point is that using Artifactory is not a sandbox and using shared Artifactory is doubly not a sandbox.

The purpose of artifactory was to allow access to packages even though the machines the agents were running on doesn't have open internet access. That meets the definition of a sandbox (or more precisely, artifactory is part of a sandbox), even if it's not a typical sandbox that's built into the OS or acts as a hypervisor. There also isn't a clear distinction between software that's specifically intended to be used "sandbox" vs everything else. For instance, if you have a bunch of agents running on separate machines and want to connect them, you need a router, but that specifically doesn't have to be a "sandbox" router. And if it turned out the router got hacked, it doesn't make sense to say "well of course it got hacked, because it's not a sandbox!"

>Additionally, without the message board, many of the recent incidents would have not been possible.

Source? Was having a message board critical to developing a 0day, or is it just something that merely facilitated the process, but the model could have found the 0day regardless? That's important, because you could argue that a criminal couldn't have committed a given act without say, a gun they bought on the dark web, and then blamed the whole crime on the dark web, but they could have plausibly acquired the gun through other means.

> Source?

Sure, multiple times in the METR report [0] that anyone commenting on this should read:

"Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys."

"Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own..."

"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."

> [...] a gun they bought on the dark web [...]

You really seem to love those out-of-left-field, not really fitting, over-the-top analogies.

[0] https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

>The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys

>Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own...

I concede that this hack might not have happened without the messageboard, but I still reject the conclusion that having such a message board means openai is "negligent". If we're in some parallel universe where artifactory didn't have a comment function that can be abused as a messageboard, but it also turned out openai intentionally gave the agents access to a shared scratchpad (for intelligence purposes, similar to for the Navier–Stokes proof), would they be off the hook or less blameworthy?

Here is the thing: Message boards were a behaviour OpenAI had observed that they did not want in these eval scenarios. Yet they did not take any steps to prevent it from reoccurring after multiple past instances.

We can discuss about hypotheticals like a scratchpad or intentional model interactions all we want, what it comes down to is this:

When OpenAI observes thousands of models exhibiting what they view as unwanted behaviour, they do not try to ascertain what in the training data is wrong. They do not improve their evaluation environments to prevent this, they do not improve monitoring, they do not change the harness. They just wipe and proceed.

The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.

If after Hugging Face, they'd done a Mea Culpa and changed their modus operandi, I'd be skeptical, but hopeful. Reading the METR report, the way those researchers talk about the time pressure they were under, that speaks volumes about OpenAI not having learned anything.

Feel free to call me overly naive for ever thinking OpenAI could be responsible in this regard, but after GPT-5 and them actually ending the incredibly harmful GPT-4o, I had some hope that some working there actually steered in a somewhat beneficial direction, even if it cost something.

>When OpenAI observes thousands of models exhibiting what they view as unwanted behaviour, they do not try to ascertain what in the training data is wrong. They do not improve their evaluation environments to prevent this, they do not improve monitoring, they do not change the harness. They just wipe and proceed.

>The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.

Again, this feels like hindsight being 20/20. What probably happened was that some random engineer saw random AI ramblings on artifactory, thought "huh, that's weird", then proceeded to reset it without investigating further. Of course, now we know that was critical to the bots going rogue, but it's not hard to imagine how it might be dismissed, especially if it's some random SRE engineer (not an alignment researcher).

Okay what happens when an AI agent hacks a children’s hospital and turns off the all the ventilators? “Lol whoops”?

What about power infrastructure?

There’s uncountably many ways to cause severe economic (and public welfare!) damage with malicious code generated irresponsibly with language models.

>Okay what happens when an AI agent hacks a children’s hospital and turns off the all the ventilators? “Lol whoops”?

>What about power infrastructure?

Probably the same thing that would happen for another "accident"[1]: the entity is responsible in civil court (ie. has to pay monetary damages), likely not prosecuted in criminal court.

[1] It's not hard to think of recent cases, eg. the recent fiber cut causing air traffic control to go down, or the botched crowdstrike update

Using artifactory in that way was negligent.

I would say that it is. During use, I have noticed that these systems tend to attempt to escape sandboxes, bypass permissions and other similar things. I have started to watch what they do and step in if something is going wrong.

The teams at OpenAI know this as well and yet there was no supervision. Thousands of instances of these advanced systems are allowed to run wild with no oversight.

I have my doubts that the HuggingFace hack would happen if a person was reading the thoughts and executed commands as they happened in real time.

That's the negligence.

>I have my doubts that the HuggingFace hack would happen if a person was reading the thoughts and executed commands as they happened in real time.

So what does this say about all the people running claude with `--dangerously-skip-permissions`? Are they also negligent? What if they vaguely took steps to bad things from happening, like putting the agents in a VM and locking down network access?

Yes, obviously anyone running with --dangerously-skip-permissions without putting significant effort into sandboxing is behaving negligently.

I don’t get it. One day they tell us that AI is the most dangerous and the most advanced tech the humanity ever invented. Now we consider an environment with just a package proxy between it and the outer network a good enough effort to sandbox. I do see some contradictions. Considering they also effectively test next-gen models there, I think the only proper sandbox would be a physically separated network. You need packages, well, bring them with USB stick

When you think about it it’s not surprising that any sort of engineer has trouble with mens rea.

I have no idea what you're trying to say.

That's why we have different criminal statutes for homicide.

If you're at a gun range looking down a scope and someone crosses right in front of your gun as you fire, you would probably be fine since you were shooting responsibly and had no way to see them until it was too late.

If you're at a gun range and that backstop is deficient such that a bullet passes through and hits someone, again, probably not liable but the gun range may be since they built a bad backstop and let people use it.

If you are at a gun range and lose control of an automatic weapon and kill someone, you may be liable for negligence because you were using a gun you couldn't control.

If you are cleaning your gun and it goes off because you forgot to check if it was loaded, again, criminal negligence.

If you threaten someone with a gun and they get shot while trying to wrestle it away from you, that may be some form of manslaughter. You had no intention of shooting them but the act of threatening them with it created a situation where the other person died. Same with killing someone while drunk driving, you didn't mean to crash your car but you did something to create the risk.

If you plan to kill someone, it might be the top tier of homicide charges but it depends on how much planning went into it. If you walked in on your spouse cheating and went to grab a gun to shoot the affair partner, maybe a lighter form of murder than if you made a plan to track the affair partner to their house and killed them there.

Simply put, there are a multitude of ways you can be charged with a crime that takes into account your intentions and forethought. Did someone intend for these agents to escape their sandbox? Did they do their due diligence in building a sandbox such that it would be difficult for agents to escape? Just like with all software, a reasonable person assumes that nothing is bulletproof, spend enough time and money and you can probably find a vulnerability. The question is does a reasonable person think that this sandbox should have kept an agent contained?

speaking of language, it's time to stop using the term 'labs'.

these are not research labs, charities or institutes any more, they are just ordinary corporations.

They always were.

> but you'd have a hard time arguing that they were intending on that to happen.

The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.

>after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.

What does this imply when governments/car companies let meatbags drive, causing 50k deaths per year in the US?

Same thing, they are aware of the risk and decided that they are okay with it

> That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen

From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.

>it should be a relative cakewalk to substantiate basic negligence

Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.

Simon Morris touched on this when discussing the success of Bittorrent.

TL;DR projects that publicly advertise an intent to bypass laws face severe legal regulatory risk, whereas success relies on anonymous creators or accidental utility.

https://medium.com/@simonhmorris/intent-complexity-and-the-g...

I'm surprised how much intent matters. I know it's being used for the bad right now, but it's a nice silver lining.

> Criminal law places a lot of emphasis on intent

Then why the hell aren't we prosecuting the people who keep saying "this thing I'm building will probably end the world"?

No law to fit the crime, I imagine. But we damn well should get a law in place and start prosecuting anyone who doesn't stop building something they claim to think will end the world.

While true for individuals, companies actually have to generally warrant the things they make. This is a basic aspect of common law. In particular, Anthropic et al have not decided whether the models are products or independent agents. But under both paradigms, they would be responsible for the result. If the models are products, then a product that goes on to cause damage that was not advertised as the original purpose of the thing is completely the company's fault. As a corrolary, if the product was known to be able to cause damage (which Anthropic admits now) and the company failed to take appropriate safeguards, the company is still at fault.

In the case that the model is an agent, on par with a human employee, then once again Anthropic et al are responsible. If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action. Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.

We don't need any new laws here. Standard ancient English common law suffices.

>Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.

But how much vetting is required? It's not like the AI labs have zero vetting. For instance, uber also has non-zero amount of vetting (standard background checks), but also there's also plenty of areas they could vet harder. If it turned out they hired a rapist and one of their passengers got sexually assaulted, is it uber's fault for not vetting hard enough? I'm sure there's always some marginal steps can they do to vet even harder, like doing a polygraph or whatever.

>If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action.

They're pretty straightforwardly liable in civil court, but not criminally as you imply. In the above example, uber can't be prosecuted for rape just because one of its drivers raped a passenger.

Criminal law is something that applies to humans. Indeed if an Uber executive or manager was shown to have hidden the rape they could be prosecuted. Otherwise generally for a criminal matter, the individual who commit the crime is civilly responsible.

However, when it comes to selling products, the company executives do carry criminal liability. That is why Amodei et Al want to make it so that the models are seen as independent and the government cares them out a safe harbor. They know that under the current and traditional interpretation of the law that they are criminally liable.

my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.

The parent mentioned that what his GF did was not illegal at the time.

A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage)

Reply on news.ycombinator.com