> Even if it was a mistake or "mistake", the part where the agent can exploit things across multiple levels like that, isn't just marketing.

When we say "safety" people do not think we are protecting them from accidental automated crime at scale being committed on their behalf.

It is a marketing stunt in the sense that, instead of being honest and saying "Taking structured output from token predictors and running that as commands for external tools, then passing the output back to the token predictor in a loop can lead to very bad consequences, especially if they have internet access.", they say "Our models are so freaking smart they can hack HuggingFace"

But aren't there plenty of uncensored/unrestricted models out there? Where is all the collateral damage?

Also, I think if Claude and OpenAI are just doing industry standard guardrails that everyone else is doing including DeepSeek, the fact that they are talking about it more than other companies makes it part of the marketing campaign.

As an analogy, if Apple were to talk up their phones having fast charging but their charging speed is the same as everyone else (or slower).

>If you have access to a SOTA model without guardrails, provide a prompt that lets the agent come up with "creative" solutions to problems, and don't properly isolate it, they can end up inadvertently hacking 3rd party companies.

it's pure delusion to think that's a SOTA specific quirk. DS/GLM/K3/Qwen/Claude/GPT/Gemini/Grok will all break CFAA laws with clever prompting, and they'll do it well if given the harness and tools they need.

This is evidenced by a huge uptick in game hacks and reverse engineering articles, some even featured on this site.

the reality is that it doesn't take a superintelligence to do something against ' the law ' , and 'being hacked' varies from victim to victim.

Will Phillips consider themselves hacked when a clever user prompts an AI into getting their toothbrushes to dump rom? Is it 'hacked' to clean-room re-implement a video game net protocol in order to produce private servers?

Judges opinions vary.

> If you have access to a SOTA model without guardrails, provide a prompt that lets the agent come up with "creative" solutions to problems, and don't properly isolate it, they can end up inadvertently hacking 3rd party companies

You can do the same with improperly-managed human interns (see for example, the big AWS outage caused when an intern pushed a firewall rule directly to production), so I'm not clear what the big deal is here.

Yes, the AI may be faster/more-knowledable than an intern, but the threat model is exactly the same as for a rogue employee.

And it's always SOTA models.

We've had gpt2 intelligence for a while. Where's the end of the world?

Fake it till you make it?

The boy who cried gpt2 agi.

Of course it was. They clearly decided that the benefit to the company valuation was higher than the potential downsides when announcing to the world that they committed a criminal act via negligence.

If it wasn't a marketing stunt, they would have at most quietly settled any legal matters with huggingface behind the scenes, fixed their evaluation harness so it wouldn't happen again, and avoided the potential future liability.