For many of the new bugs, do we think they would all have been prioritized in the past? Are these all critical bugs that would have all been addressed in a timely fashion or are they getting done because its easier to do.
Another way to ask it would be, do we think we're discovering that Chrome had more big holes than we thought or are we raising the security bar by fixing smaller holes?
To me the most probable explanation is that they automated a way to find (and fix) existing vulnerabilities in a way that was not possible before.
Some holes were probably very small, some were probably almost impossible to actually exploit, I don't doubt it. But still, I find it very hard to not consider this a strong security improvement overall (unless they made those numbers up)
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
Linux Kernel: https://lore.kernel.org/all/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80Kzs...
The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older developers are less likely to be offended, having to deal with this as a matter of course (as the mind declines).
And either way, what, we are going to keep this line going for another 5 years? Aren't you bored?
The citation was in support of the post above mine and was incidentally a link to a mailing list. I did not read the mailing list threads out of personal interest, admittedly. I think it's a particularly bad way to communicate (took 15 years for me to figure it out), so I avoid them.
> Like even in that linked thread, is personal offense like you lay out here
Taking it personally, is a concrete demonstration of what I described. The replies to my comment, are unsurprising.
Lol! What about fuzzers, linters, typecheckers and formal tooling? There’s plenty of machine code evaluators that people do use because it’s better than relying on human skills.
The issue is the actual report and the lack of information.
Pretty sure Chrome has been using all of those forever, along with some of the best security researchers in the world, yet AI (which is what GP really means by “machine” here) is finding way more bugs. I think GP’s point is that AI makes some people uncomfortable because it operates more like a human than a special purpose tool.
As for lack of information, I’m curious what you’d be interested in seeing. More information about the kinds of bugs it found perhaps?
https://cacm.acm.org/research/lessons-from-building-static-a...
This is the kind of report that you can reflect upon and learn from instead of feeling like you’ve just read a marketing piece.
I am very skeptical of these workflows, but I also use LLMs regularly. I specifically use them because they are better than me at sifting through massive amounts of complex information. They are also quite, uh, sketchy, for things that are significantly easier. A total mixed bag in my experience that ultimately is useful, and I will continue to use
Like having real difficulty with basic counting while being able to solve extreme math problems.
Being context machines, talking about what they’re definitely good at and definitely shit at, in broad terms, has been… difficult.
sure, moreover - maybe big AI usage significantly influenced the amount of bugs. So, the picture can be like that: - 2025: 50bugs found, 45fixed - 2026: 500bugs found, 450 fixed
I previously reported on my own and was ignored; now, not so much. I assume that the assistance by the LLM is providing needed detail or formatting. (Yes, I am reading what I am submitting and making sure it is optimal before sending it.)
And what has changed now is that the higher ups at Google do indeed see a business interest in fixing bugs and giving the credit to AI to sell us more AI.