Google fixed more Chrome bugs in June than over the past two years, thanks to AI
blog.google
blog.google
It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability.
C and C++ are simply not fit for purpose when large scale software projects are concerned. All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem.
The hundreds if not thousands of developers working on Chrome weren't idiots who didn't know what they're doing. The complexity of programming in C/C++ is simply beyond most intelligent individuals' ability to get perfect all the time.
There are a number of challenges that make browsers more challenging (even in memory safe languages like Swift and Rust).
1. Back references/pointers like `parentElement` to other objects in the graph that create dependency cycles (where traditional/simple reference counting will prevent the objects being deallocated).
2. Interacting with (and creating resources in) a garbage collector when running/evaluating JavaScript code.
3. Just-in-Time (JIT) compilation of JavaScript and other complicated interpretation-compilation pipelines that can allocate and transfer objects between the different stages.
Maybe Rust let's you encapsulate the unsafe a little better than C++ but the wins are going to be surprisingly small. If you had to build it today you'd maybe use Rust but the case for rewriting an existing C based runtime in Rust is not that strong.
Also for things like interpreter loops the absence of computed goto in Rust stable is a real performance killer. Explicit tail calls (the `become` keyword in nightly) work but maybe you don't want your big rewrite to rely on that.
This isn't really a fair take. Keep in mind that C/C++ has been the backbone of the most important software in the world since the 1970s. At that time we didn't have virtually unlimited compute and memory at our fingertips the way we do now. It was a huge improvement to have a high level language which still could be optimized nearly as well as assembly.
Its staying power is due to the fact that we really haven't had many good alternatives until maybe the last few years. Things are not going to get ported and rewritten overnight, especially when they largely just work like Linux & PostgreSQL. We'd nuke a huge amount of institutional knowledge and inertia to change those up.
And C/C++ are still key in the embedded space where we still have devices constrained on compute and memory. That's not going to go away anytime soon.
This criticism feel like someone complaining about why anyone still uses horses on the day the Model T was announced. Change takes time.
I always remind people that C is a systems programming language and not fit for regular application programming. For that you need an application programming language like Pascal or Java.
The programs that do benefit from speed aren't file browsers.
I also recall having my shell crashing (zsh or bash, I don't remember).
I don't quite remember the details of the shell crash, I only remember it being on a tool that've never seem crash due to a fault signal. It was somewhat amusing.
This is simply not true. Guardrails are enforced by the compiler, and both gcc and clang have a myriad of flags to make C safe, not to mention Fil-C.
For example, I kinda know that the code
for (size_t i = 0; i < 64; i++) {
// ...
}
This relationship breaks down a bit with a lot of higher level languages where what you want to occur is higher level. This can be both a blessing and a curse; it can enable easier opportunities for optimization but may also may make automatic things downright pessimizied with very very very large effort and expenditure to get things close to usable.
You really do have to write the code that iterates over a list and applies a transformation, you can't just do
[x + y for a.x, a.y in b]
Take, for example, Ghidra. It has a feature to decompile blocks of code into C(++). Other reverse engineering tools follow suit. Even in C, it's very possible to model assembly instructions that don't have a 1-to-1 mapping to a particular language construct as a function call (and this is what these do).
But, there is a reason that, even with all of the architectural differences in the world, whether it be x86, x86-64, PPC, ARM, RISC-V, Xtensa, MIPS, etc, that you can provide a mapping that goes in 1 way (from source to object code) and also the other way (from object code to """source"""). It'd be much more difficult to do that with other languages, especially in an idiomatic fashion.
That doesn't mean we shouldn't do it.
One surprising thing is that, in the grand scheme of things, developing an alternative like Rust is not that expensive. It takes something like a small team a few years. Maybe a person-decade of effort to get to a viable point? Maybe two, whatever. The industry as a whole throws away orders of magnitude more engineering-years than that on vanity projects and internal dysfunction every year.
Of course, even once we have an alternative, the switching costs are high. And the barriers are far more social, organizational and political than they are technical. But even so, moving to alternatives incrementally has been viable for decades. And hey, if we include moves to garbage collected languages, we have been moving quite a bit... but we could have been moving more. It was only a matter of will and, upstream of that, cultural change.
So really, the problem isn't that people complain too much, it's that they don't complain enough :)
Yet they seemingly don’t mind that it practically guarantees them to make blunders over and over again.
In contrast, the last company I worked for did everything in typescript and the amount of problems caused by code that blocked the main event loop was staggering. And at that, our best JavaScript people generally just threw up their arms saying there wasn't a good way to fix it.
Between the two, I prefer a language that lets me fix problems.
But then again I only know because I love that shit and started out in c/embedded, c# etc. And on my own weak side I fucken suuuuck at frontend related stuff (not the design stuff at all, just...fuck you css).
These days most of the sensible people have moved from C++ to Rust and so the "serious C++ folks" are an entirely different distribution of people who are much more inclined to think memory safety isn't a big deal and you just have to not make mistakes.
Wrong. There are more than 16 million C++ developers worldwide.
The industry as a whole throws away orders of magnitude more engineering-years _daily_ in meetings discussing if scrum standups are effective.
At the core it's even more of an economic barrier which isn't even unique to capitalism.
I recall an article talking about why the Romans didn't invent steam engines and things like railroads considering people were aware that steam could be used to do work. The answer was largely that slave labor was so cheap that nobody really cared about finding more efficiency. It didn't solve any problem they had at the time.
The analogy transfers to the tech industry today, economics tend to drive most decisions. While I love technology and think research and innovation is important for its own sake -- particularly because we might not know what we're missing out on unless we explore -- but those controlling the purse strings have other priorities.
Others might point out that the economic cost of the bugs and failure cases in programming languages without memory safety might actually be greater than the cost of developing alternatives or accepting less performance in exchange for correctness. They're probably right! But unless you can measure it and present it on a spreadsheet to the management class, they won't acknowledge or understand it. It's really difficult to measure "incidents you didn't have". In any case it still might be cheaper to work on the next big startup idea, hype it & sell equity -- that strategy has produced trillion-dollar valuations recently. However correctly or incorrectly that may be, the benefit to an elite few is very real.
Wrong. It is the market that doesn't value correctness or security enough.
We have the tools (formal verification) to create bug free software (see seL4 and CompCert for example). However customers are not willing to pay for the extra cost needed.
Unironically this. Like most game developers, I was extremely sceptical of Rust, based on opinions I've read on the internet. Ugliest syntax I've ever seen, and "safety" features are irrelevant and just get in my way. Compile times are glacial. Tons of direct and indirect dependencies needed.
But then I got to use it professionally for a year or so, and I was MINDBLOWN by how much less stuff I have to keep in my working memory at all times. And libs like tokio and actix-web were so well-designed that I actually enjoyed reading the code in my free time. Fighting the borrow checker? I had like five such instances in the first month, understood what it wants and why, and it never bothered me again. Compiles were indeed slow, but I went from "if it complies the first time around, I'm probably missing something" to "if it compiles the first time around, it's most likely correct".
Turns out, I had significantly underestimated what Rust brings to the table, while focusing on stuff that wasn't really that important once you get used to it.
Nowadays I'm back to gamedev. I'm happy about that, because my year in cloud/edge dev felt like we were consistently solving the wrong problems. But I do wish my current team was more open to giving Rust a chance. It solves so many actual problems, and is being dismissed with reasoning like "sorry, I don't have programming socks". And by very smart people! WTF is this timeline?
Granted I doubt how true that is, I always understood the benefits of languages like Haskell and Ada/SPARK, it's just that pragmatically speaking I want beginners in my team to be able to code too.
I’d love to have that happen within the next century. I might not be around for it, but I’d say that humanity deserves software that has fewer memory issues.
> All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem.
Seems like hyperbole... It might be ideal if we could snap our fingers and suddenly have rust ports, but I feel like you're discounting the effort that goes into doing that properly.
Which brings me to an interesting point. We have a web engine written in Rust. It's Servo. If the Rust port is so important to do as soon as possible to avoid mayhem, why aren't companies picking up Servo and using that?
It worked well enough, maybe "fit for purpose" is a stronger word
Windows was built in C when "C was not guaranteed to win", hence why they have weird things like Pascal calling convention etc
I'll quite happily say they were never fit for purpose. C had a role in bootstrapping early software development in the absence of anything better but once Lisp, Delphi, Smalltalk and Java arrived on the scene then it, and its equally footgunnable variants, should have been pushed to the margins where hardware constraint was the primary consideration.
Writing a browser in C++ in the 21st century was ridiculous and irresponsible.
I dont think they had much of a choice in 2008. Now? Sure. I bet a rust port is in the works.
Chrome's stability innovation was to run different tabs in different processes, and the security innovation was then to sandbox those processes.
Rust would have been super nice but it wasn't an option then, and of course let's remember that Rust was developed by Mozilla specifically with the goal of making safer browsers.
Luckily one of the perks of A.I. is that it makes large scale code migrations feasible in a timely manner.
Automated tools (before the advent of A.I.) can only get you so far. The bugs being found by A.I. today could've been found by eyeballs if companies thought it worthwhile.
Did you have to scan the code looking for memory bug yourself with your eyeballs too? Yes, at every code check in you went through a paired peer code review before checking code in no matter your dev level. So two sets of eyeballs would look at every change.
So yes, we had to look for bugs in multiple ways no matter the type of bug on every single code change.
Then the testers would run all these tools and look at the output too (I was there when each org still had SDE Testers).
Did bugs still get missed? Of course, just like today even with today's new tools.
It is simply impossible, even for the smartest human minds, to manage this kind of complexity. And this is a fact.
Also better defaults.
Logic errors are "out of scope" for Rust or any other programming language.
Also, this doesn't address why the rate increased, which is the interesting part!
I'm obviously not entirely serious here, but I think this is true to some extent: the number of memory safety bugs in a codebase is finite. Once you have a way to find them, you can drive that number down to zero.
C++ has become a far safer language thanks to LLMs - at least if you run the LLMs before you are attacked.
Some of the most important and safety critical large scale projects in the history of humanity had/have manual memory management. If you think the effort of doing it correctly is not worth it for certain projects, that may be a reasonable opinion. But the idea that humans cannot write correct C/C++ programs is in direct contradiction to historical fact.
It will, as long as the code uses them correctly all the time, including when refactoring. In the end, that's still manual memory management.
For example, lot of C++ objects are tied to a garbage collected object in JS, so you need something special to hold a reference in C++ code to those, and prevent them from being garbage collected.
Another example, shared_ptr is fine if you eventually drop the reference count to 0, but in JS, you will have cycles, so the reference count will never get there. And that's one of the problems the garbage collector deals with, it checks cycles and connectivity to mark a group of objects to be released together. So you will have a special life-cycle for those objects. Another responsibility if that you can't really use the destructor there or you might trigger a delete storm and block the thread. So the garbage collector will destroy objects in batches when the page is idle no to create stuttering (jank).
I checked the stats. 73% were memory management related issues, mostly use-after-free.
The rest were logic errors of various sorts.
The world runs successfully on C and C++. Ignoring that fact and claiming that Rust will magically remove all bugs is BS^2.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
Linux Kernel: https://lore.kernel.org/all/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80Kzs...
The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older developers are less likely to be offended, having to deal with this as a matter of course (as the mind declines).
And either way, what, we are going to keep this line going for another 5 years? Aren't you bored?
The citation was in support of the post above mine and was incidentally a link to a mailing list. I did not read the mailing list threads out of personal interest, admittedly. I think it's a particularly bad way to communicate (took 15 years for me to figure it out), so I avoid them.
> Like even in that linked thread, is personal offense like you lay out here
Taking it personally, is a concrete demonstration of what I described. The replies to my comment, are unsurprising.
Lol! What about fuzzers, linters, typecheckers and formal tooling? There’s plenty of machine code evaluators that people do use because it’s better than relying on human skills.
The issue is the actual report and the lack of information.
Pretty sure Chrome has been using all of those forever, along with some of the best security researchers in the world, yet AI (which is what GP really means by “machine” here) is finding way more bugs. I think GP’s point is that AI makes some people uncomfortable because it operates more like a human than a special purpose tool.
As for lack of information, I’m curious what you’d be interested in seeing. More information about the kinds of bugs it found perhaps?
https://cacm.acm.org/research/lessons-from-building-static-a...
This is the kind of report that you can reflect upon and learn from instead of feeling like you’ve just read a marketing piece.
I am very skeptical of these workflows, but I also use LLMs regularly. I specifically use them because they are better than me at sifting through massive amounts of complex information. They are also quite, uh, sketchy, for things that are significantly easier. A total mixed bag in my experience that ultimately is useful, and I will continue to use
Like having real difficulty with basic counting while being able to solve extreme math problems.
Being context machines, talking about what they’re definitely good at and definitely shit at, in broad terms, has been… difficult.
For many of the new bugs, do we think they would all have been prioritized in the past? Are these all critical bugs that would have all been addressed in a timely fashion or are they getting done because its easier to do.
Another way to ask it would be, do we think we're discovering that Chrome had more big holes than we thought or are we raising the security bar by fixing smaller holes?
To me the most probable explanation is that they automated a way to find (and fix) existing vulnerabilities in a way that was not possible before.
Some holes were probably very small, some were probably almost impossible to actually exploit, I don't doubt it. But still, I find it very hard to not consider this a strong security improvement overall (unless they made those numbers up)
sure, moreover - maybe big AI usage significantly influenced the amount of bugs. So, the picture can be like that: - 2025: 50bugs found, 45fixed - 2026: 500bugs found, 450 fixed
And what has changed now is that the higher ups at Google do indeed see a business interest in fixing bugs and giving the credit to AI to sell us more AI.
I previously reported on my own and was ignored; now, not so much. I assume that the assistance by the LLM is providing needed detail or formatting. (Yes, I am reading what I am submitting and making sure it is optimal before sending it.)
The big problem is that AI output can be very convincing and look "right", even appear to work, until you examine it in detail and realise all the edge-cases it didn't handle.
We still have a very strong review culture, and people work hard to review their own code before making PRs to avoid wasting other people's time.
The "engineers" using that AI. Could they repass the whiteboard tests of old? Have not some of them quietly left?
> What remains to be seen is whether Google also introduced more Chrome bugs in June than over the past two years, thanks to human programmers.
> The big problem is that human programmers output can be very convincing and look "right", even appear to work, until you examine it in detail and realise all the edge-cases it didn't handle.
In my experience, I find it to be exceptionally good at exploring and fixing the edge cases. Of course the output is not human maintainable for these fixes and needs to be heavily tests controlled, refactored or just accepted as being agent-maintained going forward.
My own reviews have shifted now. I tend not to examine detailed semantics anymore. The AI is as good or better than me at assessing whether a chunk of code does what the author said it was supposed to do.
Instead my job is to spot design and architecture smells, broader semantic errors, violations of unspoken business requirements, etc.
For example, I was recently reviewing code that built out a transactional flow. Part of that flow involved recording the transaction somewhere user visible and I knew that should only happen after the transaction was confirmed. AI implemented it where the transaction was posted.
That starts as an issue of underspecified requirements but that always happens in the real world. Thus that's where I can provide the most valuable insight: assessing with that context, whether business, operational, historical, or forward looking.
Its a massive effort for our team it has to be a massive effort around the globe if you take quality series.
It will just be easier better cheaper to teach ONE LLM how to do good code review
So…are you planning to hire only seniors who already know everything? Where do these seniors come from?
That's AI for you.
At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.
No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.
I definitely make use of AI but in my experience I almost always could have done it better myself, the places where I threw AI at the problem I didn't care about the results being good, only good enough.
When we see memory and compute requirements for version x+1 of software decrease instead of increase I will happily say AI is the oracle people proclaim it to be.
Saying “never used it” or “I tried and it was useless” was literally not possible.
If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures? The general way technology is deployed is that we try to find ways to make it work, because those are the most interesting. We're not as interested in all the ways it doesn't work.
From my perspective, some people are trying to use AI in the same way somebody might use a laptop to paddle a canoe. Sure, you can do it, but it's not a good idea. The fact that it doesn't work well is not particularly interesting.
If I steelman your position, I guess the ideal repository would be a set of cases where it's known to work well and a set of cases where it's known to not work well. A little bit like ProtonDB or SteamDB, perhaps.
I would set up channels to talk about AI, encouraging both successes and failures, with proofs required for both, and punishing people who intentionally misreport on either.
That's not the way AI is being deployed. At all.
I come to HN to read about AI failures and disappointments.
Is it less than a human would have? Is it more? Is it the same type of bugs?
I could be convinced either way and it's an interesting thing to ponder in my opinion.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
Is that better testing? Or more bugs were being introduced in the first place?
I'd say that one is not really an issue. In 2012 the Pinkie Pie exploit chain already required chaining 6 bugs to lead to an exploit [1]. Since then we've seen chains requiring more than 10 bugs (!).
If you fix any one of those bugs, the exploit is non-functional anymore. Sorry out of luck.
So if, say, for every ten bugs you fix, you introduce two new ones then it's still a very net win. Unless of course it introduces a bug so bad it becomes a simple exploit not requiring a long chain of exploits.
But in the case of browsers we've only ever been moving to longer and longer chains of exploits required to pwn a browser.
A great many window of opportunities are closing for dark-side hackers / north korean intelligence etc.: there were probably exploit chains still open for exploitation in April that just got closed by Google.
If anything, besides the supply chains attacks in amateur-land, the world didn't stop working: projects (not just browsers but OSes too) are being hardened left and right.
Using AI to find potential bugs is an amazing use case and there really aren't many downsides.
> The post has counts for everything that went right and nothing for what could go wrong.
I'm not saying there aren't a few downsides but the benefits are just too good to ignore.
[1] https://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1....
1: https://www.anthropic.com/news/mozilla-firefox-security
2: https://blog.mozilla.org/en/firefox/hardening-firefox-anthro...
Previous discussion: https://news.ycombinator.com/item?id=47273854
But the fact that nobody else who made it through had an exploit and claimed $$$ on Firefox tells me that improvements were made.
And this is what blew my mind, personally; a _browser_ - huge, complicated target codebase with myriad features, many of which are 'on the internet' - didn't have any disclosures with money on the table. That's definitely a datapoint worth registering. But you're absolutely right to remain skeptical!
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
But AI is not smarter than you, AI is as smart as the user who uses it. It often makes wrong decisions unless user corrects it.
Recently I noticed an unusually high number of web inspector bugs fixed by one longtime WebKit engineer, and I suspected AI assistance, though the commit messages include no disclosure of this. Nonetheless, the increase in volume was quite dramatic.
The other day, an update to Safari Technology Preview was released, and I attempted to verify whether a couple of my bugs were indeed fixed. However, it was impossible for me to verify, because the latest version of Safari Technology Preview introduced a new web inspector bug that totally broke the features I was testing.
Thus, I remain unimpressed.
Without a doubt, LLMs have demonstrable skills and can produce code much faster than humans. I never thought that producing code fast was wise, though, even before LLMs arrived. For many years I've criticized software development based on management-driven release schedules, where developers are forced to pump out code regardless of quality, regardless of whether it's ready. Your boss may not like it, but code is done when it's done, not when your boss says it has to be done. If software complexity were predictable and reproducible, then indeed management could replace engineers with automation, but that's not how it works in reality.
Back to Chrome, the browser is notorious for continually adding invasive new "features" to the web that nobody wants except advertisers. I wonder how many Chrome bugs were the result of Google push push pushing all of this new crap on users over the years?
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
I'm not asking for a one size fits all solution. I'm asking for "vibe coding with Codex on GPT 5.6 with high is the way, but if you want to be more ivnolved, Sonnet 4.6 on Claude is the path. Here's the proof."
Saying that it depends and it's impossible to quantify allows the camp who are claiming it's more productive to say "you're holding it wrong".
I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.
I see this mindset a bunch, and I just don’t get it. Reviewing code is so much harder than writing code! To write code, I need to find one path to achieve my goal, and convince myself, the compiler, and the reviewer that it does so. To review code, I need to consider all reasonable paths to achieve the goal, and confirm that a sufficiently-good one was selected. This is why we have junior engineers write code and seniors review it. If you’re beyond your comfort level writing it, how can you possibly review whether the author (AI or otherwise) made good choices? You only have exposure to one possible choice lane, and by problem statement you weren’t comfortable populating the other lanes…
Seniors review code because they have a much better understanding of the necessary taste and discretion, and an awareness of the larger system the code will integrate into, whereas there's only so much you can learn about coding that's relevant to most of the code you'll write (and only so fast you'll ever really be able to type it, and most of the issues can be caught by automated, deterministic systems).
Also, there's code that is just tedious and repetitive to write. AI can write that and reviewing is pretty easy.
To bring back to code: They are making the point that verifying "correctness" is not the be all and end all of code review. In every situation, there are multiple "correct" solutions, and only a few of them are actually correct and fit within the past, current and future context (technical and business). If someone isn't doing this, they aren't reviewing effectively and are acting as a glorified linter, which is an astonishingly low value-add use of human capacity.
There are situations where nobody really cares about the output, and nobody wants the true form of review, so it reverts to (at best) human-linting. In those situations, engineering capability of any kind functionally has no moat and all participants are equally fungible. If the business is dependent on engineering it's also a horrendous business. I would personally avoid being involved in these environments in an engineering capacity. I mean this in a diagnostic sense - it's a terrible career move, though there will always be busy work that's exactly what we have LLMs for.
I do not know what the secret sauce is.
But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.
In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...
I've found AI to be immensely useful in gamedev, mainly UX development, research, and curation. In those areas I let it go quite wild, with mechanical and visual tests for UX, and summaries for research and curation. I've also made skills, custom subagents, hooks, and scripts (all made with AI assistance) to streamline development.
Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big. If I worked on, let's say banking, I'd be much more careful. On my day job, as a data scientist (moving towards product owner) in a large industrial company, I'm much more careful. UX can run on vibes, but anything that touches data processing or business logic needs to be looked at by a human, row by row. Depending on use cases, it may be fine to look at only the tests. The main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
It would be useful if people _ever_ explained the use cases. Even Anthropic when they're publishing their materials are hand wavey about this.
> Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big
I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
> he main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.
I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
Yes, good addition.
> I couldn't disagree more. I run an engineering team in GameDev, and AI is great at making a gigantic mess. It's great for generating a lot of code, but the minute that doesn't do what you want it to, you now need to go back and learn _everything_ that it's generated. And IME, doing that is as bad as having just written it yourself in the first place.
Yeah, I should have been more specific since our use cases are very different. I'm working solo on small games, so it's easier to keep the AI on a leash. I can imagine it can be a nightmare on large code bases with many collaborators.
> I tentatively agree here, except I've found about 50% of it is "fine", 40% of it is "this works but isn't really what I want to put into my product" and 10% is "absolutely not".
Yeah, probably our use cases are very different here too. We mostly develop/maintain dozens of microservices for internal use, mostly greenfield, relatively small in scope, and with 1-3 developers in each. And maybe 98% was a bit too optimistic. The larger and older the codebase, the more problems AI tends to create. But it's very good at helping to understand old codebases (even those written by myself haha).
Oh, and esoteric data science topics can be a minefield too. If there's a topic that's new to me, It's difficult to know what is hallucination and what is not. It's best to let AI suggest alternative methods and then study those yourself.
I include a line in AGENTS.md that says "We *always* add docstrings to methods, classes, structs, and namespaces - there should be 100% coverage of docstrings."
It will make better choices about what functions to make or remove if you force it to justify why the function exists.
And when you have to go back and understand, it becomes easier.
They try to but they don’t. As an example, we use perforce in games. I regularly have to stop an agent and remind it to use p4 edit rather than blasting through the read only flag, despite the first paragraph of Claude.md being “this is a project using perforce. Batch call p4 edit on all files before modifying, do not manually remove the read only flag”
Hi, professional game dev here. No it's not. Thank you for reading.
The fact that people jump on the "you're simply using it wrong" wagon in every HN comment section whenever someone shares their negative/neutral experience with it implies otherwise.
Sometimes you need to set your oven to grill, other times fan force. But if you're complaining about getting burned then, yes, you're simply using it wrong.
But one thing that doesn’t change is the need to specify an end goal correctly and precisely. I think the emphasis of knowledge/information-processing work is going to increasingly be placed on verification mechanisms. This is practically equivalent to precisely defining an end goal.
Spend time deeply thinking about what it means for a solution to be correct. What properties will a correct solution have? What of those properties are testable? Write those things down and tell the agent.
As models get better, agents will be able to target more and more difficult end goals. The strategy just becomes more useful. (It’s useful for people as well.)
For example, if you want an agent to write a photo editor, think about what end properties the editor should have. There are reference images for color space and rendering transformations. That’s a good start.
Sometimes the goal will be fuzzier. “I want a feature like CaptureOne where I provide a reference image and it makes my image look like that.” Well, time to think really hard about what that means. Iterate with AI on how to test for that precisely. Come up with some good metrics/heuristics. Maybe it means local contrast should match. Maybe the overall distribution of colors. Maybe something more complicated.
Then you have a target and you can let an implementation agent work against that. If it fails, it’s either because the agent is bad or your target was incorrect or incomplete. As models get better, the limiting factor becomes your ability to correctly define a problem.
Or you can also use it piece-meal and have it do parts of work that you want. It works in both ways and works really well.
I don't think there is any model right now that you can use with 0 oversight, but you can do pretty complex stuff without writing single line of code at this point.
I also encouraged my team to paste any inquiries or alerts we got in our slack channel verbatim into codex and follow that procedure until it could figure it out as a one-shot.
I still treat it as more of a design and coding partner day-to-day, so I'm typically not trying to one-shot, but it can e.g. be triggered by a pagerduty alert, reference our code, check grafana panels, query application state across different servers/clusters, and come up with bug fix PRs all autonomously, and then post its analysis, mitigation suggestions, and PR link to slack for the on-caller to review. This was all just some docs, scripts, and a little listener service to trigger a one-shot model prompt that we built.
As a design partner, it's made it easier to add more telemetry or create rapid prototypes to check my assumptions. I've encouraged my team to regularly ponder on what facts they wish they knew, no matter how difficult it would be to find out. Pretend you could just ask God anything you want to know. Then ask the AI to go add whatever's needed to find out. We should all move toward a perfect intuition for what our system is doing because all questions can now be easily answered if we can only think to ask them.
if the complexity is high but you know how the task is to be done then what you can do is divide the task into pieces yourself or with the help of the LLM and use an LLM for each piece.
if the task is complex and you don't have a good idea how it is to be done then you need to start exploratory work with the LLM (multiple sessions, don't reuse the sessions for code).
how to use an LLM correctly is a skill in and of itself. the people who say that LLM's are bad either haven't used any good ones and have such disorganized thinking processes that they are unable to encode them into English good enough for an LLM.
It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
That's my main issue with AI hypers. There's often no goal in sight, they're just busy for the sake of being busy. They talk about their process, but not their objectives. It's not about progress for them, it's only about being in motion.
https://news.ycombinator.com/newsguidelines.html
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I think a lot of us were excited that we will finally be able to polish our old software.
Many of us were never given the opportunity and were tasked to produce new features at an ever increasing pace. Depending on where you landed you can be either elated or jaded.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
However this is how it’s marketed. C-suite is telling people eventually you won’t need to read code, frontier labs saying programmers won’t exist etc.
The reality is like you said, a tool to accelerate you.
If you test each feature and have it iterate on your feedback, you can build a decent product in this way.
Does it write too much code? Perhaps. Could I bring those 100k lines through code review in a team where some members nitpick? No.
That does not change the fact that it works, and that you do not need to read the code even today.
Unless progress slows down dramatically and soon, it seems rather likely that most of us will not be reading code in the near future. Their marketing is not wrong.
It seems like when people are talking about production level quality and how the AI. should be helping them there, plenty of people comes up with their “it works on my machine” anecdotes.
I will launch it by the end of August. After ~4 months of hard work.
My effort instead goes to manual QA testing, providing feedback and preferences, and asking questions.
The limitation here is mostly that the model does not know what looks and feels good.
With improvements to the vision capabilities and a better understanding of motion, or what looks appealing to humans, the implementation could probably happen much more autonomously.
Until you go to prod, you can believe a lot of things about the state of your code. Reviewing code is not merely about “Does this things work”. Testing and linting do cover most of that. Reviewing is mostly about: Will this design cover the current set of constraints (some may be conformance) and can it evolve? Are the assumptions correct? Is the security layer good enough?…
The user interface part of the code is only one single component out of many.
Testing and linting often do not cover whether things work or provide decent UX. The 'security layer' is almost irrelevant here.
Do you do mobile app development? I would guess not.
Even though a lot of mobile app are just presentation layer for a backend service, there are indeed a few stuff that warrants carefulness.
Like any data saved offline. Any update to that offline format means a migration plan unless it’s just a cache for the online data. Then if you support something like multiple organization, you don’t want user data to jump the separation between those siloes. And finally, if you you data that are created locally, or present data that are transformed locally using some feature, you have to be sure you’re not corrupting it or open some side vulnerabilities (think about the clickable vulnerability in messaging apps).
UX and UI can be designed using a prototyping tool coupled with user feedbacks. And with the wealth of components in the mobile SDK and the UI toolkit ability to create custom widgets, it’s very easy to implement the final version of that design. Most of the delicate work is presenting the correct information and that each interaction proceeds well.
Regarding clickable vulnerabilities in messaging apps, are you referring to the exploits that were enabled by vulnerabilities in system media parser and browser components?
This has essentially no practical relevance to app implementation.
What we are left with is not corrupting user data, particularly during migrations on app updates.
Compared to the attack surface of a backend service that handles user data, this is trivial.
We agree that the delicate work is presentation and interactions working well on device.
In my opinion this is best supported by extensive manual QA and user testing, rather than code review. This is what unlocks tremendous productivity for mobile app development with AI.
I use a few of them daily. While there are still bugs, there are significantly less than there used to be back when everything was being maintained in my spare time.
Still not Netflix/Google standard yet, but definitely better than the average side project or pre-AI Bangalore special.
It works far better than the competition.
I see a lot of the disagreement here has the same form as many of the political arguments from 10 years ago.
Most of the tech people I talk to outside of HN, Reddit, Twitter, and other debate heavy spaces have a more nuanced view. In group chats we share stories about the amazing things we got out of an LLM today, or the hilarious failures or mistakes they made. We share tips and battle stories. We talk about the pros and cons of different models. We discuss the hard things we’re doing that LLMs can’t handle yet.
Go on HN or LinkedIn or Lobsters and there are some comments from people who seem like they’re coming from a different universe or timeline. Some people claim LLMs are doing everything in their life perfectly. They’re posting thought leader Tweets about working 2 hours per day and doing everything with ChatGPT voice so they don’t even need to be at their computer. Others are claiming that LLMs are useless and they can’t get anything usable out of them at all, or that they create more problems than they solve.
It’s weird. I can’t recall a time when people in tech were having hot takes that disagreed with such polarized extremes. Like I can look at LLM output all day long and see that both of the extreme takes are not right, but they’re all steadfast in their ideas.
I think LinkedIn belongs in a natural category with Reddit and Twitter far more than HN does.
Where do you suppose those misconceptions on how to use the tool comes from? That's rhetorical because the answer is obvious: the people selling the tools and their army of bots and sycophants. You can't be mad at people for being like "hey I tried to use this tool in the way it has been sold to me and its not delivering on its promises"
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.
But in the end it's you. LLMs have their limits and need humans to direct them.
If you really need something to latch onto, there's plenty of educational material available about how to be an effective scrum master. Start there.
Some AI skeptics paradoxically insist humans are irreplaceable and then get almost mad(?) when you discuss how humans are necessary in the loop. At least that's my interpretation of what is going on, it's hard to tell.
When building with LLMs I'm mostly product manager and QA, the whole task is having the judgement to evaluate and correct course. You can share advice but it's not a copy-paste situation where one set of prompts/harnesses/whatever solve all your problems forever. They have to be adjusted for a particular situation.
My problem with the “high level advice” is that the results it gives are bad. Sure, sometimes it gets it right, and it helps but I’d wager about half the time the results are just bad. Hence why I’m asking people to show their homework here. I think that the anthropic rust rewrite is great example - it shows yes you can do a lift and shift assuming you’re ok with those constraints.
> Some AI skeptics paradoxically insist humans are irreplaceable.
But yet in this comment thread we have people saying “just point it at your code and let it go” like [0], you saying “just follow the vague instructions and if you can’t get the results that I’m telling you you’ll get them it’s your problem. But it’s too circumstantial for me to be able to tell you how to work”
> where one set of prompts/harnesses/whatever solve all your problems forever
I don’t want forever, I just want to know what is actually working today, or this week, or this month.
I'm writing for myself, not including myself in any group of people.
>I just want to know what is actually working today,
With LLMs I have facilitated the writing of about a million SLOC over the past 9 months.
- A GitHub "clone" with many enhancements which all of my development is hosted on
- A task manager as though kanban had a baby with a FIFO stack
- (in progress) a CAD kernel potential Parasolid competitor
- reverse engineered a vehicle diagnostics tool to diagnose and fix some car problems I was having more easily
- plenty of other small, half baked, or abandoned projects for one reason or the other
LLMs amplify me, they don't do things "for" me. They are working very well for me, but I have definitely hit their limitations in many places and transitioned back to the hardest problems to solve being on the human side of the equation.
What they can do for any particular person depends on the person and if they can come around to really grokking the human-machine relationship.
It's not magic. Fairly obvious oversights that I or someone else on the team could've found and fixed if we had looked, but it was never a business priority so we never did until I curiously spent 5 minutes asking it to do it for me.
A few lines of Python with a time.time() call is enough to start iterating on performance improvements, and the really good models know how to use much more advanced existing tools.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
As I've seen it, there's a large chunk of those getting the most out of AI doing so because they weren't aware of or couldn't be bothered by already existing, more reliable alternatives.
I recently optimized some code asking Claude to "make it faster" 2 different programs. For each, it wrote a benchmark, gathered initial data. Emitted some hypothesis and measured data around them with profilers, then did some changes (behind feature flags), checked the output was identical in either side and through profiling that the right code path was taken, and then benchmarked both.
And it did that for multiple successive changes in both codebases. Some changes were purely algorithmic (better complexity), some were related to tradeoffs between memory and computation (caching intermediate results better), some were about creating less intermediate objects to relieve the memory pressure, some where about a better memory layout to improve data locality.
The annoying part is that some of the code that was optimized was generated by Claude in the first place, so it's a bit frustrating that it didn't do those in the first place. But I guess, my iterative approach to making those tools didn't work on big enough data sets at first where it would have been an issue.
I can see in the code when data layouts aren't optimal, and fix that.
There's a lot of optimizations that need more of a deep dive, but you can get a lot of gains by just reading and reasoning about your code/data.
EDIT: To add, there are cases were you specifically can't read code and understand performance issue, but you should first ask, is that because you just don't understand the APIs/Libs/tools you're using, or is it fundamentally difficult. For example, often at work I see people complain about their torch code being slow and needing to bust out a profiler, but often those people just don't understand how tensor operations work internally so of course they can't reason about the code and see the way they're using the lib is suboptimal.
Obviously you want empirical evidence to justify changes, but like... duh, you can read code and understand how it executes.
Keep in mind that I've never stated that all performance issues are findable through code review. Just that there are many that are.
> You can very easily spot performance issues through code.
If you think that this is false, there's literally nothing you can contribute.
Here's some slow code: https://github.com/torvalds/linux
Speed it up.
I even have an example myself, but I'm not sure I can share it in detail. It was a common idiom that was used throughout the code. After tracing a performance problem to a specific function, it was immediately obvious to me that the common idiom was the problem in this function, and how to fix it very easily. But only because I traced a performance problem to this function. Otherwise I would have skimmed over it.
I mean yeah, that's mostly what I'm saying?
The only parts where I would disagree are that capabilities are spiky, and also the fact that they're a bit prone to losing the forest for the trees without guidance (the macro/micro distinction). But yeah, by default I don't write code by hand.
Claude Code, Fable 5, xhigh reasoning, allow it to run the full CI, end to end and benchmark, it will not make silly mistakes (or only occasionally). Also, be able to state what you desire. Have any docs or materials in the same directory so the model can reference it. For even better results: turn on speech recognition and braindump what you know about the system, its goals, its context, history anything relevant, any gotchas you'd explain to a new employee or intern. Talk for 5-10 minutes. This is optional, "make it faster" can already get a large part of the job done.
And if it doesn't work well, describe what you dislike in its solution and tell it. Even just one extra iteration can make things work.
(I guess GPT-5.6 can be similarly good, I use Claude)
I feel like some people are emotionally invested in it not working and subconsciously sabotage their own effective use of the tool.
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.
One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.
They will mock things to no end. They will flat out REMOVE assertions (saying it's not needed). They can also write test to assert the wrong result.
You have to always review it, it's exhausting.
And I think I know why - neither do most people.
Claude will do some boneheaded things for sure, but it's pretty good about writing tests that are useful, and not removing or modifying tests just because they're in the way.
Claude is pretty bad about assuming that it couldn't have broken a test it didn't know about, as it has often told me "this is already broken on main" which is definitely NOT true.
Imagine you’re blind and deaf and have temporary retrograde amnesia. You “wake up” one moment with a memory of some words in your head like “what is the bug?” …but you don’t recall the context of that question, and nor can you look/listen around to observe the context.
So you don’t know whether you’re e.g. at the office, in front of your computer, in the middle of doing some pair-programming (where, yes, you’d in investigate the bug thoroughly with tools), vs. having a conversation with a colleague over lunch (where the expectation is for you to tap into your knowledge + intuitions to either guess or say you don’t know — not to pull out your laptop.
That’s what it’s like to be one of these LLMs being prompted by some agent harness. Unless the harness injects the proper context into its “recent memory”, it just doesn’t know.
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
I work on user facing applications, and since the models do not have good taste, testing the UX is essential.
If you spot a bug, usually the model will attempt to reproduce it in a new test case that does cover the actual issue.
2nd line, code review.
Do the first pass with an agent, ask it to bounce back vacuous or tautological tests. Ask it to verify that the tests verify what they claim to. Then read them yourself.
3rd line, mutation testing. If the tests don't actually catch broken code, kill the mutants.
I don't strictly mean junit unit tests.
Tests onli validate the presence of bugs, not their abscence (Djikstra).
I'll also add that tests look at outputs and don't care how those outputs are derived. E.g. code filtering the entire db in memory will be fine in tests.
Tests prove the things you thought of worked, and it often isn't hard to find the likely edge cases such that you have reasonable confidence everything works. You will be wrong from time to time, but not that often. You can prove code correct, but if the proof is wrong (common when a human is doing it), or the spec is wrong (most people have no clue how to write a comprehensive spec) it can still be wrong.
And then there's the endless code duplication, reinventing of existing code and libraries etc.
Tests =/= TDD
That'll get RL-d in, eventually.
1) verify identical query results
2) run repeatedly to get average, worst, best, etc duration of runs
Sped up so many legacy things that none of us were ever going to bother with.
Source: C. Northcote Parkinson, "Parkinson's Law and Other Studies in Administration" (1957)
I’m assuming he’s using it to mean “cruft” or “low value features” but maybe there’s some meaning I’m missing.
I originally heard it in the context of picking paint color for the bike shed, which is probably more on point for software engineers, most of whom would not be involved in any discussion around building a nuclear reactor.
People will recognize a glimmer of something they can attach to, prompt and get a few pages of junk about this narrow topic, and then contribute it in a way that further reduces the signal-noise ratio of the discussion or project.
Like, dude, we have the same tools. We can get the same slop from the tap at any time.
Can only assume english might not be your first language.
Thankfully until this thread I’d never seen anyone use the term this way.
> Can only assume english might not be your first language.
This not-subtle insult does not add anything here.
An insult would suggest it's also related to autism.
Is the harness for the AI set up with any sort of instructions to generally prefer broadly-applicable first-principals query performance analysis over unintuitive results that may be local maxima due to certain things specific to that env?
Even with humans I've had to unwind "optimizations" before that worked great for low-volume envs by taking advantage of 'be inefficient in the small scale with memory to save CPU and wall clock time' or such in a way that caused pain deployed.
Current AI isn't super effective at making the breakthroughs, but it sure is effective at democratizing the ones you can point it at.
That's democratization - shifting power and skills away from those with the wealth to purchase vulns and run offensive security teams. It's still not simple, but it's one of the reasons bigger vendors want to lock down the capabilities of models that present a threat to the wealthy and powerful.
As hardware specs get better, and model providers keep making different classes of mistakes that alienate users, more people will opt for or support local models. I tend to take Schneiers' "Attacks only improve" philosophy and apply it to damn near everything like this. Inference will get more expensive, but IMO cloud based inference is at or near the maximum that users can tolerate, across multiple spectra. There may be a marginal increase in cost, but as the different metrics between what is affordable for rapidly scalable, cloud based inference and the lag of local inference with open models and expensive hardware converge, prices will come down.
I am not the right person to say what the peak is going to look at, but when the companies with (practically speaking) unlimited compute resources and money are starting to take a good hard look, there is going to be a drive towards efficiency. I hear grumbling at work, and the appreciation from my leaders when I show up with a tool that shows how I reduced my token costs, or I keep asking the questions of what the token cost (real, and internal billing) are for tools my peers write are.
I also have absolutely eye-watering personal AI bills that are starting to make buying a better tier of local inference gear look more affordable, even with inflated hardware prices. The current generation of open models are very effective, and paying for multiple $200 dollar a month subs is not feasible longer term, and as one example, in one month, if I was paying enterprise token rates, I would have burned through $8000+ token budgets on one of those accounts, doing actual, practical useful work, it makes more sense to buy some much better hardware than to pay for one-time inference costs.
AI doesn't have enough senses yet. It's trapped in a box.
I can imagine the quality of bug fixing by unsupervised (we save money by layoffs) halucinating AI at current state of development.
"99 bugs in the code, 99 bugs in the code... Fix one bug, compile it again, 101 bugs in the code!"
(To the tune of 99 Bottles Of Beer)
But hey, one can't survive 1 week of step-by-step debugging without 99 bottles of beer.
If you want to learn darts or perfect parallel parking for example, most of the increase in accuracy comes from simply but very deliberately pointing out to your brain where you wanted to land versus where it did land.
We like to usually just do things and hope for the best. Defining success is not something we automatically do and naturally we don’t do it with AI either.
Also let me ask you why we need better and better and models if what we have already can produce good output with 'all the tooling to verify its hypotheses'
This is such a blanket dismissal that I can’t agree or disagree.
Maybe very few of YOUR problems are this way. At least mention some problem domains.
Recent experiences: compiler-related (helpful), UI-related (agree it isn’t testable but the design iteration is quick, easy, and correct), debugging technical configuration problems (useless; I basically have to solve each problem myself before the LLM recognizes it).
> Maybe very few of YOUR problems are this way. At least mention some problem domains.
i did in second part of my comment. why do you think billions are being poured into ai if ai can already do verifable tasks.
“Good” isn’t “perfect” and even if it was, the ability to produce perfect output with all the tooling to verify its hypotheses could still be improved, in time and token efficiency, by better models producing fewer spurious hypotheses, rejecting those it does generate faster, and taking fewer unnecessary steps in confirming its good hypotheses.
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.
The SOTA: Fable, GPT 5.6 Sol, Opus 5
The "enterprise admin did not turn on the new models": Opus 4.8, GPT 5.5
The "I love hallucinated garbage": Sonnet, Qwen 3.6, GPT 5.4 mini, GPT 5.3 Codex, etc.
Results vary widely
Before that, Gemini 3 Pro in Antigravity.
I have no experience with GPT 5.3 beyond seeing the nightmares colleagues produced in their MRs with GPT 5.3 Codex. It could be that they had the distilled 5.3 Codex Spark selected, I am not sure.
But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).
I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)
That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.
that's the first mistake i made as a new claude user. i only knew what prompts people were sharing on forums. but since they had been written by vibe coders it was all complete non-sense. also think about this: software engineering is filled with highly specialised tooling that can easily improve software quality. the thing is, most developers dont know how to use them, and if they do -- using them is so slow that its almost not worth it.
any tool you can name claude can use. obscure academic proof verifiers, memory corruption scanners, debuggers, profilers, coverage tools, backdoor supply chain pattern scanners... it can run these and have results in seconds. really is like living in the future.
I've been using Claude to create a software architecture diagram. It came up with a lot of useful functionality that we had neglected to show but upon further examination a lot of steps didn't make any sense. It added a box to do validation on some data then it put another box downstream to do validation again on the exact same piece of data (the second box had a different name but upon questioning, Claude conceded that it was the exact same validation step). When doing inference on user input, it put a box to extract specific insights from the prompt and then it put another box to do inference again on the inferred data to categorize it... And this would add latency and unnecessarily strip out useful context when doing the second inference... It really only needed a single inference step...
I re-uploaded the diagram to Claude and confronted it and it conceded to all of my points and it even noticed a theme between them and suggested that the diagram exhibited "a pattern of redundant steps." But it was incapable of suggesting viable solutions besides name changes. All of its solutions made the design more complicated. It was incapable of simplifying the design. I've noticed this with AI code; it can make things more complicated, add more features but struggles to simplify things.
Also, AI is horrible at rating things... It's just too superficial. For example, if you write a huge amount of intentionally over-engineered, tightly coupled code with poor separation of concerns but you do proper linting, define all the types, add a lot of comments, it will give it a higher score than a shorter, more readable and maintainable snippet which exhibits loose coupling and high cohesion because it lacks the superficial aspects.
It's horrible because now business people who understand nothing about coding may ask AI about code quality and it will consistently rank low quality over-engineered projects that are full of bugs, unmaintable and less secure, with a higher score.
Frustratingly, if you point out problems in its judgement, it will concede to your points without reservation, even building on top of your argument... but it will never actually tell you this stuff up-front, unprompted if you don't already know it! It never seems to reveal new insights, at best it can only expand on existing insight which you've already had.
It tells you what you want to know but not what you need to know.
> others dumping their raw AI output at me
Sheesh.
I'm thinking the issue is probably that LLMs/harnesses are too easy to use? It crossed the thin line between magic and tooling and blur the mental model. If Claude Code were as hard to use as, say, ComfyUI, perhaps there would be less programmers having absurd expectation of it.
The parent commenter was trying to learn how to use the tool. Perhaps he could've used it better, but like you said, what may have worked was not necessarily the intuitive approach.
The parent comment also changed his approach, and AI worked for him to test out changes. That's not expecting the tool to do all the work, he's actually agreeing with you that "expecting the tool to do all the work" is a misstep.
Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.
Also things like making custom visualisations for comparing changes.
Give Fabe 5 access to a test database with some data, or even restricted access to your live DB and tell it to optimize then.
I've had stunning success optimizing for performance this way. In a single day, I made the core part of our app 2-3x faster.
That's like saying Fable 5 should have told me how to prompt it to solve the Riemann hypothesis like Terence Tao would. The skill of the user still matters.
"I have not spent the time cultivating the soft skills necessary to leverage this tool successfully therefore it's the tool that sucks."
How about give pointers on "doing it right?"
My expectation: inexperienced FP is trying to shoehorn a mutation-based approach into a framework that performs best with immutable data, or is manually recreating familiar structure (for loop) instead of using the idiomatic way.
But what do you tell someone in this case? They really need to learn more about the basics. Which is how I am interpreting the GGP’s dismissive-sounding comment.
Work on your project management, communication, and mentorship skills. The AI tools are much more like a team of sloppy but capable junior engineers and not a reliably consistent fabrication machine. Change your perception that one simple prompt is going to magically solve a hard problem the first time. Learn to break projects down, organize them into sub-projects, and implement them in steps. Learn to communicate more clearly, define your requirements more rigorously, and adapt around your "team's" strengths and weaknesses.
We want these things to be rigorous and flawless but reality is messy just like working with a regular team of engineers. They are good at some things, terrible at others, constantly changing, and unreliable. But we've historically built tons of reliable software on unreliable actors by focusing on process, collaboration, and communication.
AKA, not just the hard technical skills, but the soft skills that allow unreliable software teams to thrive. They don't know what they don't know and it's your job to manage that.
It's not "hard" or novel--it's just tedious grunt work. I'm happy to have AI solving that one.
Turns out folks had been copy pasting some setup code which took 2 minutes to run, and only needed to be run once per environment. Claude refactored the code to get it to run once, and since we had tests which validated the build artifacts, it was able to roll back if its change had broke the build.
I profile sql performance and LLMs find more opportunities than I could. All it takes is real data, a sql repl and an agent. Just ask the agent to use the repl to EXPLAIN and profile the sql. It works amazingly most of the time.
Once I have the baseline I'll just set a goal to improve performance by 10x. Works better than it has any right to. Have to be careful to exclude changes that massively increase complexity for tiny gains after its done, but the big improvements are often sensible choices that I'd also make as an engineer (i.e. more efficient data representation, caching and memoization where it matters, parallel processing, etc.)
I guess just another area where the LLM is useful only as long as you remain in charge using your own programming experience as a guide.
And at worst you still saved a bunch of time on writing the tooling to test the issue
As an example Opus 5.0 is in completely different class compared to Cursor Grok 4.5 even if the benchmarks don't show such massive difference. Not even talking about regular stuff like Sonnet or Composer or stuff like that.
For performance testing, I wrote isolated testbeds that try to impair the system in realistic ways (latency/jitter/bandwidth limit on logical WAN hops when load testing), and Fable is happy to send a bunch of agents at it and iterate until it gets the results it's looking for.
I think that if you are used to Sonnet medium or something, this will surprise you, but models like Fable and Sol on high/xhigh will really dig deep until they meet your goal. (I mostly use this for bug hunting and not perf, but ... I think it can do perf if you set it up right.)
1. Ask Claude Code or coding agent to research the internet, documentation, and Github for examples and learning working with similar problems.
2. Make the ten best examples of solving the problem based on the research.
3. Run each against the database enough times (100, 1000, or 10,000,000 depending) to empirically prove which is optimized. You can set other criteria based on your knowledge.
4. ???
5. Profit
The point is, it is cheap to research combing through 1,000s of examples and learnings and test the best and most relevant.
Every single message from him, literally every time he communicates, it's a massive wall of text, overflowing with scope-creep suggestions like nothing I've ever seen before. It's impossible to ask him a simple question and get a simple answer.
And this seems to be a broader trend with AI in general. It's getting more and more difficult to retain some semblance of brevity with AI tools. I'm constantly asking them to "be brief," and "only answer the question I asked," and "don't provide more information than I requested," but they keep churning out more than I want in their responses.
It looks like a subtle attempt to use up more tokens.
For a couple examples, working through an animated loader for html/js/css with an svg for the org. Another was working through a library implementation to work against an interface that was designed for Mongo, but the org is using SQL Server. Latest was a quick util to extract a zip file of pdfs into a 1bit(b/w), zopfli compressed png file per page.
Generally stuff I could do, but would take me a few days for research and experimentation vs an hour or two with AI.
Not sure how efficacious but I know the reviewer is just relying on the AI decision.
If you have to lie to get your tickets fixed, that sounds like either you're bad at explaining why things are important, or you're working on things you shouldn't be.
That really bothered me that I finally announced AI "code of conduct" (more like AI usage expectation) and stressed that if one is not so kind as to have his/her coworker review AI output as it is, then I'll take formal measures to address that. We'll see what happens in the near future.
Really worrying that some people are totally comfortable with relying 100% on AI to do their work. Then what's human agency? What's the point of having them in the team (as opposed to just building an AI agent and work with that)?!
There's never been a shortage of low-value coworkers because hiring is hard and firing is unpleasant and painful in many ways.
But there's a weird phenomenon with AI where a lot of people are using it to actively call attention to the fact that they aren't doing anything but call the tools.
Rarely in the past have the I-just-want-to-coast-doing-the-minimum folks called so much attention to it!
Apparently, it takes a lot of time and work to manage all those AI chat bots.
You seem to presume that they realize they're calling attention to it.
On Stack Overflow, there was this phenomenon for a while where people would blatantly copy and paste answers from ChatGPT, have them deleted, and then run to the meta site to complain. The complaint would either be in barely comprehensible broken English or in recognizable ChatGPT-ese. It would involve swearing up and down that no LLM assistance had been used. Looking at the user's history would reveal a gap of several years of not answering questions, and then the old answers would have the same barely comprehensible broken English (and often not be very good technically, either). The OP demonstrated a genuine complete inability to understand how obvious the ChatGPT use was.
This happened multiple times with different users.
(and yes, i've experienced this exact phenomena as well)
For code gen and review, it's been pretty good since recent Opus versions, and I really like Fable's outputs (though a bit costly in terms of use). Fable is much closer to my own style for the couple things I've done with it.
I have yet to even consider replying to emails/messages with AI. I sometimes write a wall of text on my own, but it's entirely me.
And with hiring effectively frozen there is not even an incentive to let go of those people because they will not be replaced - they may have contributed little overall (routine tasks, oncall weeks) and now contribute 10x more spam while before they would not even try to investigate an issue, but we don't have the cycles to allocate that to someone else. And every time you tell them something they try to gaslight you by telling you they're trying to learn and you're being too aggressive.
I've become so frustrated by seeing claude-isms in slack or in emails it has become impossible for me to use it to work on my personal projects and I'm moving over to codex. Maybe it's time to try pi and some chinese models, definitely I need a break from this madness.
My experience is if you have been racking up the tech debt and have some low hanging fruit, Claude is really good at cleaning it up. If you have been properly stewarding the project, it's much less effective.
If you give the LLM the ability to run queries against prod-like data and review the paths, you have a fully autonomous system that can correctly optimize your ORM/queries. Been doing this for months.
pgAnalyze is a great complementary tool for this and they offer MCP
I had worked on some earlier deployment environments for a few models where we focused on one version of Triton and one runtime technology. Even upgrading to a newer Triton version was brittle, involving a lot of command-line changes at various phases. This was written mostly pre-Claude-getting-real-good. I decided that probably Claude had matured and was way better at understanding the particulars of AI-model-GPU-deployment-and-technologies. I worked with Claude to make the framework a much more lightweight wrapper, ignorant for the most part of a lot of the deployment internals.
After this refactoring and doing the first couple of models, it's quite amazing at how well Claude can figure things out. For any new model we now set up its "specialization" directory and its documentation subdirectory, point Claude at the proper AI-model files, point Claude to the sample non-optimized inference code and test data, and point Claude to a similar conversion we've done in the past. There a multi-layer class hierarchy dealing with various tiers. I ask Claude to explore the existing conversion/packaging, the model, any documentation that comes with it (a lot of times there are unexpected twists), the sample code, and the desired multiple use cases the model is meant to address, and the test data. Claude has been trained, I'm sure, on a lot of AI model conversion, so it's able to synthesize the full multi-stage conversion/deployment pipeline, come up with appropriate test cases for all the use cases involved. There are usually between 3 and 10 refinements after initial synthesis, fixing outright errors, refinemnts that the data-science team requests after playing with test deployments, etc. The options and pitfalls are vast, and without Claude each preparation likely would 10x or more longer. I just put most details in Claude's hands, and make sure the general framework is good enough to provide external uniformity. When all is working, it takes a couple of hours to make sure the documentation is good.
All this to say that at least for this domain, Claude / AI has been a game-changer and has sped up the process amazingly.
> Added support for model interoperability to leverage the unique strengths of both open-weights and proprietary models
Because creating 100x more bugs and fixing 100x more isn't something to be proud of.
But it's an open source project, you can go and figure out whether your assertion is correct.
They obviously have the full git blame statistics but chose not to include them in the blog post, which is a bit concerning.
That's not a great argument. GPT3.5 was launched over 4 years ago, so that's already significant compared to Chrome's age. LLMs notoriously also produce more verbose code than a human, so its not out of question that it might produce 5x more bugs on average given the same time frame.
HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.
The elephant in the room is how many turtles obtained law degrees.
The elephant in the room is whether this is worth discussing because we're all in a simulation.
No, it isn't the elephant in the room. You made up some horseshit that betrays how little you know about the field, which will get upvoted because HN has a hearty contingent of "if I pretend AI is actually useless, maybe that will manifest in reality" zealots. It's bizarre.
If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.
As if humans are incapable of spotting critical bugs simply because they've existed for a long time? And we know humans don't produce fresh bug-free code, because the old bugs exist.
This is wrong on almost every count.
1. The numbers are by milestone not month.
2. The latest milestone (or month) did not exceed the previous two years of milestones.
3. The only thing discussed are security bugs, not bugs in general.
What is true: the article is about Chrome.
Google Home became so unbelievably buggy, and I felt so betrayed by that, that I stopped using every single Google product - from Gmail to Android to Google Cloud Platform. The few physical Google devices that I'm yet to replace account for 10% of my tech, but still cause 90% of my problems.
Google.
We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack. All decision-making processes are here subjugated to what fits Google's adEmpire. This is a perpetual system of control amplifying abusive systems in place. We already see this with the recent age sniffing; Google just yesterday announced that all android users must hand over their age to everyone else. Before that google kicked out or locked out open source alternatives to android (or, at the least, made their life significantly harder than before). These are not isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one tyrant company dictates so much of people's open lives (and no, Firefox is no alternative; Mozilla is basically a nerfed and bribed entity that has given up on firefox many years ago already; ladybird may become an alternative at one point in time, but right now is not and there are questions over how Kling handles the overall project, but these complaints are significantly below what Google is doing globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
We do / have? Multiple even. Of course, the vast majority of consumers doesn't care so they don't get as much traction.
You're absolutely right! Let me fix that.
On the other hand, my experience so far has been that they are terrible code-reviewers, to the point almost everyone ignores any kind of generic LLM review entirely (correctly - most of it is noise).
I feel the industry obsession with full-automation causes them to go down the wrong rabbit-holes on full-automation vs augmentation. In my view would be significantly more effective if the LLM review process in forge-tooling was designed with a human reviewer in the loop. The goal imo should be drastically improving the review efficacy & throughput of the human element, who needs to be acquiring confidence in & socializing the change in any sane org anyway. For example iteratively & interactively rubber-ducking out analysis + remediation, or interactively providing focusing guidance on the patch, instead of as agentic batch or harness processes as is cool nowadays. Sometimes a custom REPL is just better for an expert system. These things matter more as more and more code is LLM generated.
First they made it so when it came to rest it would then start scrolling again and overshoot by 2 lines before coming to a halt.
They fixed that after a few weeks then it worked until last week and now it’s left in a state where as it’s coming to a halt it jerks every time like dropping frames of the scroll almost.
I know it’s not my machine, config or trackpad because I’m seeing it on multiple computers.
Any tips?
Then try starting with voice mode (if you're comfortable chatting out loud) and just talk your way through it.
Build a sandbox, download Codex CLI or Claude Code or whatever and spend some time doing some creative stuff with it just for the sake of learning how it all fits together.
Pay attention to the inputs you provide and the outputs that they result in. Keep your bullshit detector engaged: Bots often lie.
If you get stuck, or it gets stuck, or you want prompting advice or whatever: Ask any frontier-level bot for help. Sometimes, it's very instructive to get help from Claude for an issue with Codex, or vice-versa.
Want better tools? Ask the bot to suggest some that exist. (None of the existing tools fit? Have the bot write new ones.)
All of this stuff is always in a state of flux, so even with the lies they'll do better at teaching than any fixed reference will. They're LLMs and processing language is what they tend to be best at... so go ahead and use that.
And remember: They're designed to behave kinda-sorta like humans, but they are not humans. They're just computer programs. If you don't like their output style, or they don't like your input style: Ask them how to implement rules that make them knock that shit off. :)
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
Google claims to help open source developers, but donates millions to the Alpha Omega Foundation that did not give access to Mythos to Curl and only found one issue.
The Alpha Omega people appear to be selected on physical appearance and take the money away from open source without doing much.
The entire Chrome article is obviously directed at suits and unsurprisingly hypes AI for better promotion chances.
> the “latent security issue.” Code that is safe and robust in isolation can be transformed into a critical vulnerability by an entirely unrelated, minor logic change elsewhere in the tree.
Similarly, you can accidentally fix a bug without discovering/triaging it, so the pointless repeated bug lifecycle is incomplete
Might as well be that the learning here is that if you put resources on fixing bugs, you can in fact fix a lot of bugs.
But since Google cares mostly about its investors, the numbers and a mention of Gemini in such a blog post are more important.
Some were probably just theoretical "if we have over 2B of this item, it'll crash because X/Y/Z and that can be exploited", some are probably in rarely used features.
Once the LLMs are done going through the code, the baseline will be all better and they will probably fix a lot less than they have now. Or they will have LLMs automatically check every single crash bug that has been submitted and take care of those too. I am assuming that they are probably already using this as an input source for their AI.
Just to have a sense of proportion, considering an average software engineer salary of $200k, that same money would buy you 650,000 engineer-year salaries. Of course there are other expenses, but that gives you an idea of the order of magnitude.
I'll believe in AI when I get to resize my Configuration Manager!
https://developercommunity.microsoft.com/t/Resize-configurat...
Just open up a configuration window manually, tell if that the Window is open and it can pull apart the executable from there.
Biggest impediment would be Fable safeguards if it tries to decompile.
And introduced how many new bugs? thanks to AI ...
Do you import new more bugs?
Fast foot didn't kill fine dining, but it is killing us..
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
There are many things it's not good at (eg design and architecture), but that doesn't invalidate the things it is good at. Everyone is learning how to best use it and there are growing pains. Ignore the hype and try to find the common patterns in what people say.
And it wasn't just 2 humans but 2 Googlers. One Googler author and one Googler reviewer works. But an external author needed 2 Googlers to review changes, and all files needed to be reviewed by one author at least.
So if you touched multiple files owned by different people for a more complex feature, you would naturally get there.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
0 - https://www.lacksan.com/updates/ 1 - https://github.com/Lacksan-Dev/HP-ZBook-Performance