Comment by simoncion | Hacker News Reader

Comment by simoncion | Hacker News Reader

Parent Full thread

> Predictably the discussion is already veering towards OpenAI's negligence...

> Cherry on top: That was part of an attempt to jail-break itself via self-prompt injection.

We continue to see so-called "prompt injection" "attacks" in the wild that override a user's intended program with an attacker's [0], and/or the LLM producer's intended "safety" instructions with the user's. The fact that this sort of program hijacking is possible at all is strong evidence of negligence. Why?

OpenAI and Anthropic both claim that they're working on very dangerous Internet-connected tools. So very dangerous that the production of and access to said tools needs to be tightly regulated, they claim. If one actually believes that the computerized tool one is working on is very dangerous, one generally doesn't design that tool so that it blindly executes instructions handed to it by complete strangers on the Internet. That's akin to connecting the sole activation switch for a biosphere-evaporating firebomb to the Internet.

The major LLM producers are so obviously negligent and -as a bonus- have openly admitted to committing cybercrimes [1] that would get people like you and me fined out the ass and jailed for ages if we did them. The tragedy is that they're making so much money for the rich and powerful that -much like the architects of the 2008 housing crash- they'll never see any meaningful punishments for their actions.

[0] One recent example is <https://agentic.tracebit.com/context-bombs/>, but there are so, so many more to choose from.

[1] ...the "cyber" prefix is so stupid...

Yes, prompt injection is an issue with model safety, which is what we should be focusing on. I meant to say the discussion about OpenAI's negligence in securing agents and their infrastructure is the red herring.

That said, nobody has been charged for these hacks despite openly talking about them because typically you need to show intent. If intent was not a requirement, they would have been in trouble way back when the first AI-assisted suicides happened. Lawsuits have been filed, but OpenAI's whole schtick is "these agents are so dangerous because they do all these crazy things without being asked to."

As far as we know nobody told the agents to do any of this, or even that it's OK to do this. If someone can find any proof of anything approaching actual intent, I'd bet there would be no shortage of attorney generals willing to be build their career on this case. After all, there are already many AGs investigating OpenAI.

> I meant to say the discussion about OpenAI's negligence in securing agents and their infrastructure is the red herring.

It absolutely is not. It's yet more evidence that the culture inside these companies is entirely inadequate for a company that's building what they appear to be claiming are WMDs that are very likely to be species-ending.

> ...because typically you need to show intent.

a) You seem to be suggesting that criminal negligence doesn't exist. You also seem to be claiming that deploying and operating computer software that you built [0] that you don't just know but widely advertise has a "discover and exploit faults in someone else's computer systems" feature without ensuring that that computer software cannot access other people's computer systems isn't -when viewed in the most lenient possible light- incredible negligence.

b) Go look up the facts of weev's case. weev's intent was very obviously benign and prosocial. The only reason he didn't spend four years in jail and have to pay tens of thousands of dollars was because of a choice of jurisdiction error made by the Federal government.

[0] "You" in this case refers OpenAI, Anthropic, and other major LLM providers. Don't bother with a "But what if the people running the software had nothing to do with building it!" retort.

Of course criminal negligence is a thing but you should look into the bar required to prove it. For one, it requires the highest standard of proof. Then there are at least 4 very specific elements that need to be proven, and it would be interesting to see how that could be achieved without the benefit of hindsight, given that we've seen no technology like this until just 4 years ago.

If there was a case to be made for criminal negligence, I would say it would be with the AI-assisted suicides, and lawsuits are pending but we've not seen anything come out of that yet.

> Go look up the facts of weev's case. weev's intent was very obviously benign and prosocial.

Oh I'm very familiar with the case, which is why I know the investigation revealed detailed chat logs where weev and his accomplice discussed multiple ways to illegally profit from the breach, including shorting AT&T stock, selling the data to spammers / phishers, or organizing a spamming / phishing operation themselves.

I am very curious about what sources led you to associate the words "prosocial" and "benign" because weev is very, very, very clearly a noxious, antisocial person who spends a lot of his time harassing people. I would recommend you feed "weev criticism" into a Google AI overview. Random quote from his wikipedia:

> In early October 2014, The Daily Stormer published an article by Auernheimer in which he effectively identified himself as a white supremacist and neo-Nazi. He is known for his "extremely violent rhetoric advocating genocide of non-whites", according to the SPLC.

And as you said, he got off on a technicality. That case was heralded as a triumph of justice purely because it followed the rule of law despite the obviously unlawful intents of the defendant.

Reply on news.ycombinator.com