We have a bounty program. If you can find security bugs in Firefox, please let us pay you for them. You don't need to provide a fix; a testcase that crashes in an interesting way is often enough to qualify.
873 karma · joined August 30, 2019
So the best answer I can give is: I dunno, maybe it's possible to find bugs like this using Opus, but if so, where are they? Did nobody think to try "please find the bug in this code" pre-Mythos? I've done enough auditing with Opus to be convinced that it can be a good assistant to somebody who already knows what they're doing, but in practice the big wave of AI-discovered bugs started with Mythos.
I'm sure lots of people have assumed they could send a publicly available model bug hunting and find things. I have not noticed a huge amount of success. We've had some very nice correctness bugs reported, but skimming through the list of security bugs I've fixed recently, the AI-related ones all seem to be Mythos.
My best guess is that Mythos is just enough better along just enough axes that its hit rate on finding potential bugs and filtering out the real ones from the hallucinations is good enough to matter. Like, there's no obvious qualitative difference between 3.6kg of uranium-232 and 3.8 kg of uranium-232, just a small quantitative increase. But if you form both of them into spheres, only one of them has reached critical mass. Can you do something clever to reach critical mass with 3.6kg of uranium? Maybe! But needing to do something clever is a non-trivial barrier in itself.
I did get Opus to do an audit for similar problems elsewhere, to supplement the investigations that we were already doing by hand. It initially thought it found something, but when asked to produce a testcase, it thought for 20 minutes and admitted defeat. I suspect that the difference between Opus and Mythos is in small edges like this: if Mythos is smart enough to spot why Opus's discovery didn't work a little bit faster, and it can waste less time chasing down red herrings, then it's more likely to find a real bug within the limits of a context window. It's not that Opus completely lacks some capability, it's that it has trouble chaining all the pieces together consistently.
In this particular sense, AI tends to find bugs that are closer to what we'd see from a human researcher reading the code. Fuzz bugs are often more "here's a seemingly innocuous sequence of statements that randomly happen to collide three corner cases in an unexpected way".
Outside of SpiderMonkey, my understanding is that many of the best vulnerabilities were in code that is difficult to fuzz effectively for whatever reason.
If you look closely at, say, this patch, you might get a sense of what I mean (although the real cleverness is in the testcase, which we have not made public): https://hg-edge.mozilla.org/integration/autoland/rev/c29515d...
In general, I would say that our use of "vulnerability" lines up with what jerrythegerbil calls "potential vulnerability". (In cases with a POC, we would likely use the word "exploit".) Our goal is to keep Firefox secure. Once it's clear that a particular bug might be exploitable, it's usually not worth a lot of engineering effort to investigate further; we just fix it. We spend a little while eyeballing things for the purpose of sorting into sec-high, sec-moderate, etc, and to help triage incoming bugs, but if there's any real question, we assume the worst and move on.
So were all 271 bugs exploitable? Absolutely not. But they were all security bugs according to the normal standards that we've been applying for years.
(Partial exception: there were some bugs that might normally have been opened up, but were kept hidden because Mythos wasn't public information yet. But those bugs would have been marked sec-other, and not included in the count.)
So if you think we're guilty of inflating the number of "real" vulnerabilities found by Mythos, bear in mind that we've also been consistently inflating the baseline. The spike in the Firefox Security Fixes by Month graph is very, very real: https://hacks.mozilla.org/2026/05/behind-the-scenes-hardenin...
I kept waiting for "sea of nodes with CFG" to be shortened to SeaFG, and it never happened. I guess maybe it's ambiguous out loud.
For a variety of reasons I don't think this particular approach is a good fit for a JS engine, but it's still very good to see people chipping away at the design space.
Copy-and-patch is a technique for reducing the amount of effort it takes to write a JIT by leaning on an existing AOT compiler's code generator. Instead of generating machine code yourself, you can get LLVM (or another compiler) to generate a small snippet of code for each operation in your internal IR. Then codegen is simply a matter of copying the precompiled snippet and patching up the references.
The more resources are poured into a JIT, the less it is likely to use copy-and-patch. You get more control/flexibility doing codegen yourself.
But see also Deegen for a pretty cool example of trying to push this approach as far as possible: https://aha.stanford.edu/deegen-meta-compiler-approach-high-...
The prose isn't polished enough to be AI. AI generation is unlikely to produce missing spaces like "...which are not readable to humans.SDB uses eBPF ...", or grammatical inaccuracies like "Ensuring Fully Correctness".
As for the data race thing, it seems to me that there's a pretty clear distinction between rbspy's approach (as described in reference 1) and this blog post. rbspy is walking the native stack, which occasionally fails. SDB seems to be looking at Ruby's internals instead, and has some sort of generation-number design to identify cases where there was a data race.
Beyond that, this post just absolutely sounds like what somebody would write if they were trying to describe in prose why they think their multi-threaded code is correct, especially the "Scanning Stacks without the GVL" section.
The main justification for CacheIR isn't that it enables us to do optimizations that can't be done in other ways. It's just a convenient unifying framework.
asm.js is more like a weird frontend to wasm than a dialect of JS.
It gives us a lot of flexibility in choosing what to guard, without having to worry as much about getting out of sync between the baseline ICs and the optimizer's frontend. To a first approximation, our CacheIR generators are the single source of truth for speculative optimization in SpiderMonkey, and the rest of the engine just mechanically follows their lead.
There are also some cool tricks you can do when your ICs have associated IR. For example, when calling a method on a superclass, with receivers of a variety of different subclasses, you often end up with a set of ICs that all 1. Guard the different shapes of the receiver objects, 2. Guard the shared shape of the holder object, then 3. Do the call. When we detect that, we can mechanically walk the IR, collect the different receiver shapes, and generate a single stub-folded IC that instead guards against a list of shapes. The cool thing is that stub folding doesn't care whether it's looking at a call IC, or a GetProp IC, or anything else: so long as the only thing that differs is the a single GuardShape, you can make the transformation.
Here's a long-standing SpiderMonkey bug: https://bugzilla.mozilla.org/show_bug.cgi?id=894971.
Here's a JSC equivalent: https://bugs.webkit.org/show_bug.cgi?id=224077.\
Both of those bugs (especially the JSC one) sketch out possible solutions and give some insight into why this is hard to implement efficiently. In general, it adds a lot of complexity to an already complicated (and performance-sensitive!) chunk of code.
(https://searchfox.org/mozilla-central/source/js/src/vm/BigIn...)
Take the fizzbuzz example with a missing bounds check. Rust can't prevent you from generating JIT code that omits a bounds check on an array and reads/writes out-of-bounds. The sandbox doesn't prevent out-of-bounds reads/writes, but it guarantees that they will only be able to access data inside the sandbox.
This means that logic bugs in the JIT compiler are no longer immediately exploitable. They must be combined with bugs in the sandbox implementation. The article's claim is that, unlike compiler bugs, sandbox bugs tend to be amenable to standard mitigation techniques.
This article isn't dismissing the value of memory-safe languages. It's identifying a problem space where current memory-safe languages can't help, and providing an alternative solution. Currently, every browser JS engine is written in C++, in part because Rust doesn't solve the big correctness problems. If the sandbox approach works, then using Rust for other parts of the engine becomes more appealing.
It's easy enough to rewrite this C++ code to do the right bounds checking. Writing the code in Rust would give even stronger guarantees. The key point, though, is that those guarantees don't extend to any code that you generate at runtime via just-in-time compilation. Rust is smart, but not nearly smart enough to verify that the arbitrary instructions you've emitted will perform the necessary bounds checks. If your optimizing compiler decides it can omit a bounds check because the index is already guaranteed to be in-bounds, and it's wrong, then there's no backstop to swoop in and return undefined instead of reading arbitrary memory.
In short, JIT compilation means that it's ~impossible to make any safety guarantees about JS engines using compile-time static analysis, because a lot of the code they run doesn't exist until runtime.
The speedometer graphic was inherited from Speedometer 2. When Speedometer 2 was released, scores were in a reasonable car-speed range. The combination of hardware and software improvements meant that early versions of Speedometer 3 (which includes a subset of Speedometer 2 tests) were consistently scoring above 140, so we adjusted the scaling factor (IIRC, by ~20x) to give plenty of room for future improvements.
If you run all the tests in half the time, your Speedometer score will double. If your score improves by 1%, it implies that you are 1% faster on the subtests.
(There are probably some subtleties here because we're using the geometric mean to avoid putting too much weight on any individual subtest, but the rough intuition should still hold.)
Benchmarking is hard. It is very easy to write a benchmark where improving your score does not improve real-world performance, and over time even a good benchmark will become less useful as the important improvements are all made. This V8 blog post about Octane is a good description of some of the issues: https://v8.dev/blog/retiring-octane
Speedometer 3, in my experience, is the least bad browser benchmark. It hits code that we know from independent evidence is important for real-world performance. We've been targeting our performance work at Speedometer 3 for the last year, and we've seen good results. My favourite example: a few years ago, we decided that initial pageload performance was our performance priority for the year, and we spent some time trying to optimize for that. Speedometer 3 is not primarily a pageload benchmark. Nevertheless, our pageload telemetry improved more from targeting Speedometer 3 than it did when we were deliberately targeting pageload. (See the pretty graphs here: https://hacks.mozilla.org/2023/10/down-and-to-the-right-fire...) This is the advantage of having a good benchmark; it speeds up the iterative cycle of identifying a potential issue, writing a patch, and evaluating the results.