Hacker News Reader

Comment by melvinroest | Hacker News Reader

Wow, this announcement is good content marketing.

Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.

It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.

> We then ran AISLE's autonomous AI system against curl.

They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.

View on HN

The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.

Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.

Also sounds incredibly compute intensive.

Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.

This sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.

You also quickly get into diminishing returns with fuzzing. Generally a bug is either going to be found relatively quickly with a given fuzzing approach or it's going to be nearly impossible to find. You're usually better coming up with more intelligent fuzzing approaches than you are just dumping compute into it.

Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.

Back when I first heard of it, the HuggingFace hack reminded me about how you don't have to be particularly smart to find vulns, just aggressive in looking for them. AFL on its own "learned" how to construct well-formed JPEGs and PDFs by fuzzing decoders for these formats, back in the pre-LLM dark ages.

Of course the HuggingFace hack proved to be much more than that...

It's an effective approach. Google project zero started doing this in 2024

https://security.googleblog.com/2024/11/leveling-up-fuzzing-...

Their system can run with various models, they go into more details in this article.

https://aisle.com/blog/system-over-model-zero-day-discovery-...

It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.

>All six are rated Low severity

This says it all. Nothing important was missing. This is marketing hype.

> Wow, this announcement is good content marketing.

Why do you think companies hire PMMs?

> what models the use under the hood

Presumably their own, wouldn’t they?

You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.

Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.

Default to gpt 5.4 nano

https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...

It defaults to gpt5.4 nano

https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...

A repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.

I am pretty sure that the nano-analyzer is just a limited open source demo of their "System over Model" thesis from https://aisle.com/blog/system-over-model-zero-day-discovery-..., not the main product.

Reply on news.ycombinator.com