Hacker News Reader

Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).

I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.

OpenAI said:

> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.

It was a test of raw capabilities of the underlying model.

Alignment isn't alignment if it can be turned on and off at the whim of company employees.

This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.

Alignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.

There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints

A way to help prevent some of that catastrophic damage, is to make companies accountable for what their AIs do. A major problem with AI companies is that they like to point to the AI, as if they're minimally involved innocent bystanders, when that's the furthest thing from the truth.

It’s a shame the target was HF. If it had been a large bank or other institution we might get to see this play out in courts.

I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.

I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.

I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.

A model that can do that is aligned with me.

The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)

Even AGI doesn't give you a model that can read minds and forecast the future.

But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)

That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.

I don't know, but these were exactly the questions the industry had to handle with CORE Impact and Immunity Canvas, and ultimately all the way back to Dan Farmer's SATAN before that.

The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.

It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.

If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?

> It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.

I mean, if you train models to complete tasks, then you shouldn't be surprised when they do crazy things to complete tasks.

As a (somewhat) less serious example, Claude code absolutely adores grepping for credentials to complete tasks. I was building a RAG app and it literally went looking for my API key to make the tests pass. Obviously I stopped it, but this is textbook RL issues, the model finds an easier way to get the reward, so it does whatever it takes.

As humans we are also vulnerable to that class of attack. A blue amazon vest and some boxes would get you into most buildings

With a high-vis vest and a clipboard, you can get almost anywhere. Almost.

I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.

So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.

Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.

When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?

In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.

Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")

> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.

The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself while online.

I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.

Do you want this supermodel to cause harm whilst finding these vulnerabilities?

it seems pretty likely that openai was asking the model to do something bad, and the model did something different thats also bad

if the operator wants something bad, an aligned model should execute on it

This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).

The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).

Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.

Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.

Is turning the alignment features off more like multiplying "don't be evil" by 0, or by -1?

Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.

AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.

Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.

Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.

Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.

But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.

I dunno man, the original announcement said things like:

> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy

And

> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers

Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?

I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.

Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!

Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).

You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.

Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].

[1]: Opus 4.6 was used for https://www.noahlebovic.com/testing-an-autonomous-hacker/

Thanks. I’ll concede the point and update accordingly.

Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.

I think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.

Also, that’s the wrong mental model. Anyone who runs a sass platform or a website knows that the Internet is already full of millions and millions and millions of bots and scripts and other random shit that’s always trying to attack you, often completely randomly. Security is always a battle between good and evil. All I can say is that if you’re in charge of keeping something secure, you should probably try to get your hands on the best tools to do that. I think the problem in this case is that the best tools to do that are also the tools that are making it more easy for evil to occur. So it’s more for me a perception of someone creating a threat and the tool to fix it and then charging for it which doesn’t feel right.

That's exactly what they are doing. Out of all the civilian applications that AI capabilities have, why has this surfaced as a priority for demonstration? Probably because of money.

> a priority for demonstration

1. AI companies do make defense against threats a primary thing they sell to companies, including because companies want it - should AI companies not do this? (And of course this kind of thing isn’t what AI companies demo to consumers.)

2. AI is also tested to see more of its worst case in terms of security - but the tests are supposed to remain sandboxed. (OpenAI is explicit that this was an OpenAI failure that openAI must fix.) But beyond such fixes, should such sandboxed testing not be done?

https://arxiv.org/abs/2605.11086

If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.

Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!

Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.

And that’s all just “from memory” without something like RAG or web search access…

If anyone wants to run the guardrail-free Qwen models, here’s a link to the download plus a script to run it: https://news.ycombinator.com/item?id=49002475

Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.

I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.

Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).

It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.

It’s simply revisionist IMO to claim this stuff was latent all along.

This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.

Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?

Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.

Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".

> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up

There are multiple AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.

Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).

There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.

Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.

The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).

Simon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it access to packages via the internet, instead of self-hosting them. Apple was hacked by an ex employee to download stuff for openai. Now hf is hacked by openai, and we only have their word that this was all a miraculous accident.

I think it's an interesting story and probably really does say something about the tenacity of the new OpenAI models. I just don't think it says the thing about security that breathless coverage claims it does.

Openai hacked HF with a zero-day. Definitely an interesting story! I just find their explanations hard to believe. They're admitting to a great deal of incompetence. I know, don't ascribe to malice... but still, the more extraordinary the claim, the more evidence is needed to defend it.

Edit: News just in. 'Be skeptical of OpenAI's claims' - The Guardian https://news.ycombinator.com/item?id=49038060

re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)

I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.

An upside is that it means really really good scanning will probably be cheaply commodified if it hasn't been already by one of the new AI pentest startups.

The real victims will be pen testers, who work a job whose economics was already really tough.

How's VFIO on Firecracker coming Thomas?