Agentic coding notes
danluu.com
danluu.com
> In general, when I talk to software folks about testing, I'm coming from such a different place that they immediately look at me like I'm an alien, so let's talk about how we tested at this hardware company I worked for, Centaur, which informs my biases about how I like to work. Some of the things that we did that were or are unorthodox in the software world are:
> Hired dedicated QA / test engineers, with testing being a first-class career path on par with being a developer - No code review by default - Virtually no hand-written tests - Constant testing via what programmers sometimes called property based testing, randomized testing, fuzzing, etc., although we just called those tests (hand-written tests were called "hand tests"). - Large regeression test suite (3 months wall clock to execute on compute farm) - No unit tests
Anybody here tried that (or a similar) approach? Especially going all-in on property based testing and fuzzing with no unit tests.
I tried that approach somewhere before and the initial results were promising, but ran into political issues so the idea was canned.
This is a blast from the past. Centaur was doing VIA Technology's CPUs, and I was at VIA (in Taiwan) while this author was in Centaur (in the US). I was on the embedded side, but I remember some distinct collaborations with the US team, so there's a non-zero chance to have crossed path.
I undrestand for fuzzing you have a very basic "doesn't crash" metric. Property based tests.... you gotta write properties for the PBTs to work on. What is the randomized testing hitting?
I noticed map tiles were not working and started to tell it, but then all of a sudden they reappeared and checking the logs it had found the issue and autocorrected itself.
The key here is feedback loops and systems annealing.
As for Dan, my God I love this guy. Glad someone posted it!
This is most likely because we take similar approaches towards things.
>For no particular reason, I've always liked designing experiments and measuring things. This was true long before I ever thought about careers and it's still true today. As discussed here, measurement is one of the primary themes of this blog, maybe the primary theme. Lucky for me, this lifelong hobby has been something I've been able to make a career out of. And even luckier, this skill seems to have been made relatively more valuable by coding agents3.
>3.And, coincidentally, as with testing, it happens to be a skill that gets developed a lot more at CPU companies than in typical software companies, even ones that produce highly performance sensitive products, like databases. Above, we estimated that the effort spent on testing at the CPU design shop I worked for was maybe a ~2:1 ratio over what you'd see in a traditional software company. When it comes to benchmarking/evals/experimental design, the denominator is low enough at traditional software companies that it's hard to estimate the ratio, but it's surely at least 10:1 and 100:1 and 1000:1 are plausible numbers as well.
>Of course, by focusing on and developing much more expertise than software companies in these areas, chip companies are often relatively in the stone ages in a number of other areas. Relatively speaking, I'm also relatively weak in most of those areas. I think this works out ok in the context of a company, where it's valuable to have people with complementary skills, but it can definitely cause some problems in interviews. [return]
Notice how much numerical difference Luu attributes to highly performance sensitive software products versus a traditional software company.
As for Claude and Codex in mid-2024:
>At the time, I didn't find this useful enough to use for anything where I knew what I was doing, but it enabled me to embed a little web game into that post and do other tasks that would've required me to learn something about an area where having actual expertise will probably never be particularly interesting to me, such as building a web app.
Me neither when it comes to web apps. I always thought the fundamental thing with the greatest room for improvement was getting the most out of stand-alone electronics, before it made as much sense to even network them to begin with. Looks to me that performance progress was less than halfway there when it got to be reversed in personal computers, and now more resources than ever are being put to keep personal computers from allowing users to get as much advantage as they could have been. Basically more effort than ever imagined since before the 1990s, toward a return to centralized data centers instead, not much differently than we had with mainframes. The difference is that "everybody" has a "terminal" now but those user PCs are going to need to become less-capable as stand-alone personal computers, and more like "dumb" terminals otherwise a growing number of high-rollers will not be able to fulfill their massive vision as overwhelmingly.
So what if I thought it was fairly intuitive learning to program on mainframes back in the 1960s. I was pursuing natural science not computer science, in ways people like Wozniak and Gates were not. For me the programs are supposed to be a tool, and one that's still not always necessary to achieve the optimum outcome in the research lab. It took a number of years before I was able to pay for the experimentation by doing chemical testing in the lab, and was basically the only computer pioneer in my field for a while after that, as alternative labs also had some emerging "computerized" or "digital" instruments but not for user-programming.
Anyway, by the tine the 1990's rolled around, with all the overtime at the bench, it gave me about the equivalent of 40 years of intense testing in only 20 years. From where I have never stopped, but it was mostly chemical testing. Not as applicable to things like CPU design as it could be, but the hardware I build has to handle chemicals not only electrons. Usually quite toxic materials, hazardous in other ways, and highly regulated.
And I was even more out-of-date by then on computer languages. Much more of a disadvantage compared to digital hardware engineering like where Luu is coming from, since I was not the coder those folks were, and they don't have the time to put most of their effort into the kind of code that software engineers have to do.
I didn't code every year, just as needed, and "shipping" code was not an objective at all. By then I had founded my first company and I made more money using my code than I was before, or developed routines that made the work easier. The LOB was mostly scientific and automation. I figured anybody would prefer NASA-type performance if they could, so why not, I had no deadlines, and these are some risky chemicals.
It should be easy to see that the only unfair advantage I had was a lifetime of testing up the wazoo, even if only a baby part of that was testing code. And I already could tell that testing had to be at least a bit lacking in the commercial software that had appeared (and is still often raising its ugly head). As a total laboratory system I had always been more reliable than that and I was not ready to stop at all.
Ended up at 100:1 in testing:development ratio to hit the sweet spot. That's not relative to any other companies, just what it took to give lab business clients their money's worth. Of course lots of "unit tests" and code reviews were included. With chemicals you must leave no stone unturned even more so than plain electronics. Never could have done it if I wasn't already giving clients their money's worth without the code.
Otherwise the likelihood of unforeseen regrets is far too high for mission-critical application, and can not be very accurately estimated either.
The whole time it was on my mind how I would incorporate AI into some advanced automation, if PCs got powerful enough, but that was a bit of a dream 30 years ago. AI was just unheard of for any type of everyday use. I had to accept that if I ever did want to ship some software I would have to be more of an architect and let professional engineers rewrite my stuff in a more modern language that they have experience in.
Now with things like Claude and Codex I could be doing it all on my own, but nah. I've already got a well established lack-of-momentum and a couple years ago it looked like LLM autocoding was impending so I figured it would be good to see how AI develops from the sidelines. While I'm past "retirement age" anyway. Why would I settle for my own single-handed code when now I could have a team of software engineers who are using AI to help them. Just like I always figured I would need a team to do all the coding in order to make a "product" out of my efforts since before the '90s.
If I get such a wild hair I'll still have to do all the testing myself regardless :\
I always get the impression from using hardware and other anecdotes like this that it is rare for hardware companies to know how to do software development well because their core competency is hardware. In fairness, it is uncommon for software companies to know how to do software development well.
Every form of testing has its value when done well and they are using several forms that most software developers don't use- probably helps make up for the lack of code review and unit tests. But if they incorporated code review and unit tests their software would likely be even higher quality.
Property based testing is amazing, but it won't provide full coverage. Regression suites are amazing, but generally the most expensive form of testing in terms of time to write and maintain tests and time to run them.
Today AI can crank out unit tests so its silly not to have them.
Certainly it is very uncommon for to be reviewing a postmortem thinking ‘I wish we were more thorough with code review’ rather than ‘what changes to our testing practices could have increased our chances of catching this bug’. I don’t think the value of code review is normally in catching bugs.
That is a massive amount of information even if we are being sloppy with it. You can read The Hobbit and the first Harry Potter book cover-to-cover and still have room to spare. I would deeply struggle to develop a world model this detailed for any business. Anything that needs to get more specific than these narratives can be a SQL query tool into the data warehouse, grep over the codebase, MS graph API lookup, etc.
Giving the business a balanced way to collaborate over this one shared model of the world is a new challenge I am beginning to engage with. I've also noticed that the world model will compound on itself in terms of self-detection of update opportunities. The more constraints there are, the more likely we appear to violate one.
If only. There is a huge difference between "Gives good responses/can easily spot things within N context size" and "Technically works but sucks within N context size", almost all models basically become cave-people once you go beyond 50% of the "supported" context size, meaning while they may technically work with 1 million output tokens, those last 500K tokens are gonna be massively "dumber" than the first 500k tokens.
https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...
I haven't even begun to try to comprehend how to use fuzzing testing to improve the ability to find bugs, but it sounds really interesting. I've seen mutation testing to be very useful for finding gaps in tests, so I can only imagine that fuzzing + LLMs might produce insane results.
You should talk to https://www.mechanize.work/ for sponsorship/credits and about environments.
> If a company were shipping bugs at, say, a hundredth the rate we were at Centaur while relying primarily on review to catch bugs, then I could see their point, but that's not what's happening at the typical software company where people don't want to move away from human review [there might be non quality (as in non bug rate) related reasons to keep human review, such as keeping a high bar for code quality or keeping the codebase human-understandable, which pretty much immediately stops being the case if you let a fleet of agents go wild on a codebase] because of the perceived risk of shipping bugs.
I don't agree that the main point of a code review is quality assurance; there are other reasons that have more to do with team convergence and junior staff learning: https://www.embeddedrelated.com/showarticle/807.php
I work on large C++ applications used by international airlines. If this software failed, it would make national headlines.
Claude Code with Opus 4.8 is great at handling the boring code I don't want to write myself. It gets it right almost every time.
However I still review every change and test everything before committing.
Trust, but verify.
This blog is quite unreadable for 27/32" monitors.
Reader view makes the text too narrow: https://imgur.com/a/yQqzxco
Sure i could take extra steps to make it more readable, but at that point I rather not read it as im not that interested nor invested.
Totally fine, no complains from my side. But If the author wants to increase the reach of their posts they just might use agentic coding to have the llm optimize the website for readability.
Only complain would be that its sometimes difficult to see the parent comment(s) when you scroll down.
HN could fix this quite easily by making the current parents sticky while you scroll.
Even with it's issues, the latest models are going to disrupt the labor economics.
Who cares about AI, I wanted to read about living in Galapagos
You're going to need a big big solar panel to run local inference during the day, but luckily there is plenty of sun!
https://science.nasa.gov/earth/earth-observatory/dry-tortuga...
You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed. I feel like a babysitter.
> You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed.
Eh, that's just because it's the current frontier model. Give it a few weeks, and prices will drop.
Like with Uber and Lyft, the low prices were a fight for market share, but now they have successfully captured that market share the focus changes to balancing their books.
I make that prediction, because the people who pay for subscriptions but only use them moderately at best are truly profitable.
The subjective anecdotes from HN users matter because they are not data and are much harder to game. Not impossible to game, always be aware of users with low karma, but more difficult than gaming a benchmark.
Let's take this [1] benchmark. A bit more context here [2].
Here models are asked to create kernels for running inference on models. This is a benchmark perfectly suited and highly relevant right now. It's easily verifiable, an active are of research, and the results are immediately useful.
Say you have 1 unit of compute, it costs 300k $ and serves 1x users. In comes Fable and after one session it gives you 30% speed-up on your 1 unit of compute. It can now serve 1.3x users. How much is that one session worth for you? How much is it worth for a company using 10 units? 100 units? How much is it worth for a hyper-scaler running 10.000 units? How much is it worth for a lab that trains the next frontier model and then serves it from 100.000 units? 30% is relative. And the cost for one session is really meaningless. It can cost 1m$ / session and it would still be worth it for someone.
[1] - https://kernelbench.com/mega
[2] - https://x.com/elliotarledge/status/2072814573753975266
Anthropic says the change is about capacity and is temporary. In its launch announcement on June 9, 2026, it says:
"After this point—when sufficient capacity allows us to do so—we aim to restore Fable 5 as a standard part of subscription plans. We intend to do this as quickly as we can."I wouldn't start with Fable - when I use burndown loops I tend to include instructions to document progress and set aside anything that turns out to be harder than expected, and solve the easy stuff first. When a model runs out of easy stuff and start struggling to make progress on what is left, I can let it keep churning on that - they get there eventually - or I can bump it up to a smarter model if one is available.
Opus had churned a week driving down spec failures, and did a great job. The 150 Fable took overnight were the ones Opus had kept putting aside.
Oh well, it was pretty funny, all things considered.
Config -> Switch models when a message is flagged -> false
That should stop it entirely when a message is flagged and then you can come back without Opus having potentially made a mess.