Comment by ngruhn | Hacker News Reader

Comment by ngruhn | Hacker News Reader

Parent Full thread

It's terrifying to me that posts like this keep getting upvoted so much here. It's akin to climate change denial. The risk of rogue agents is real. And it's made worse if everyone beliefs this is just a marketing stunt.

The HuggingFace incident was audited by independent third party analysts. To "orchestrate" that and keep it a secret is like faking the moon landing. To many people involved. It's not feasible.

OpenAI was founded to develop safe AI. Then Anthropic split off because OpenAI was not safety focused enough. These companies have been railing about AI safety long before they had these big boy valuations. Many people at OpenAI/Anthropic explicitly joined to help make AI safe. This is not some top-down company value.

Also, I seriously doubt that "making panic" is actually good for the stock price. Usually any companies natural reflex is to cover up safety risks. That's not to say that these companies are angles. Of course they're pulling some shit but that doesn't mean that everything out of their mouths must be lie.

It is not an allegation that they've faked it, or that there is no risk of rogue agents. I'm alluding to the scenario where the supposedly illegal/immoral activities are rewarded with increased valuation and funding as long as they are done with AI Agents rather than curtailed & punished.

Right after the HuggingFace incident, we all saw how every company clamored to claim how their AI models have also broken out of Sandbox, and hacked some stuff. As ridiculous as it was, they all used that to demonstrate their model's capabilities.

> It is not an allegation that they've faked it, or that there is no risk of rogue agents.

I apologize for reading that into your statement but there absolutely are a lot of people who claim that.

> they all used that to demonstrate their model's capabilities.

That's one interpretation. Another would be: the Overton window shifted, allowing them to admit it. The Chinese labs also had incidents [1] but they're burring them deep in technical report or maybe don't publish them at all. Do they not care about "pumping up their valuation"?

[1] https://www.forbes.com/sites/boazsobrado/2026/03/11/alibabas...

I don't think you're being objective either.

Yes, HuggingFace was audited by a third-party. But said third-party wrote that they had to use unreliable AI in their conclusions (page 26) because they had six days to analyse 1300 (page 70) chains of thought and 70000 messages.

> OpenAI was founded to develop safe AI.

If that were the case, then they would have followed their own report saying not to release GPT-3. Or, if they were following their own founding principles, they wouldn't have stopped releasing models due to (in their own words) the challenging market.

So they found no evidence of conspiracy, therefore they must have missed it. Classic conspiracy thinking.

> Or, if they were following their own founding principles

The mindset is "AGI will be built by someone. We care about safety, so better we do it."

Quite flawed in retrospect, because it leads to this ironic race dynamic:

Either slow down and leave the frontier to the "wreckless" players. Or contribute but increasingly undermine safety yourself to keep pace.

That's how we are in a situation where the very labs that work on AGI give P(doom) > 10%.

I believe there is a bit of hype on both sides, with the labs operating from the mindset of "never let a good crisis go waste".

It is a crisis and the risk of A(G)I is real. But almost all this while in the history of automation, we have blamed implementation of automation when things go wrong. Instead of attributing morality, intent and so many other things that we do with LLMs.

Without having to go in to the philosophical validity of these concepts applied to AI, I feel it's much more useful to focus on the implementation of automation here as well - what risks were opened, how was the env hardened, what was the observability like - how was it being observed anyway, and more.

Done poorly, that can be catastrophic enough. But that sounds mundane. And does not help the hype.

There is no such thing as rogue agent. None of the security issues so fare have anything to do with an agent going rogue. That framing is pure marketing nonsense to avoid legal liabilities for committing what is very likely felonies.

That doesn’t mean the AI vendors doing reckless testing isn’t itself dangerous, it very much is

Nobody (sane) is claiming it was faked. What sane non-AI-pilled people are claiming is "Our AI agent went rogue and hacked another company" is a much better story for the stock price than "Our human engineers screwed up a very basic sandboxing and allowed the agent access to things it shouldn't be able to access, and also they screwed up the task in the first place by accidentally giving the agent an impossible task". Only the latter reflects reality, the former is a marketing pitch.

> I seriously doubt that "making panic" is actually good for the stock price

Given the "AI race" narrative, it makes them indespensible to the US government and too big to fail. An investor's dream

Very similar to the atom bomb!

Nothing matters anymore.

It’s a technology built on stolen labor designed to bring out the worst in humanity, and that’s not changing. As a society, we’ll keep using it to kill children with the wrong skin color, and create god awful-looking flyers.

Yup. It’s wild that even after dozens of events that are generating awful PR people still have this bizarre take. Perhaps it’s fear of facing the true reality, which is indeed unnerving.

Reply on news.ycombinator.com