How much of a grasp does one need on the UEFI code operating their computer? The drivers that operate a hard drive? The communication protocols for their monitor?
This seems to me to be the fallacy — that all code must always be understood to operate properly or to be useful. If the AI is sufficiently intelligent enough to understand the code, at some point (most) people won’t need to. Not saying that’s now, but that is likely to be where things are headed in the long horizon. That said, I don’t have a crystal ball, nor do the people claiming that won’t be the outcome, so right now it seems more like people arguing about which forks should be used while the cake is still in the oven.
But, with AI, the cost of code has gone down more than it already has. Well, cheap things are easy to throw away. So you prototype, prototype, prototype, and close the loop as much as possible with the customer. True agile development, not big A Agile.
The problem is this requires alignment from management, and we're just not seeing it at many company. They can't grasp that things have changed, and that throwing away code is free. They don't trust engineers to close that gap, so customers and stakeholders are still waaaaaay over there and we're delivering features they don't want.
Until managers give up on Agile, we'll never get time to actually write specs
To be honest, the models are getting so good that they do most of this unprompted now.
At this point if you can’t get the agent to write good code then either I) you are in a very specific niche (like Karpathy trying to write NanoGPT) that is extremely out-of-distribution, or II) skill issue, you need to learn how to prompt better.
It’s fine to have a skill gap! Just don’t delude yourself that the tools are bad and everyone claiming they are good is wrong.
This has been the pro-"write with LLMs" argument the whole time. "If you're not getting good results, you're not prompting well. FAANG are doing x% of their development with AI."
Yet all of the companies that I can think of that build on planet scale infra and have gone all in on LLM development have seen a marked drop in quality and stability since adopting LLM driven development.
So yeah, I don't buy it. When I hear that engineers are writing their with LLMs and I see quality improve or stabilize, then maybe I will start to question whether its a prompting issue.
Planet scale infra often requires some novel ideas at the architectural layer (the engineers can input those) and then it’s mostly in-distribution C++ / Rust / Go; most of the hyperscalers open-source their stacks, for one. But for two, a ring buffer, look-aside cache, deterministic hash, b-tree, lsm-tree, etc. are all well-known patterns.
Hyperscaling is also often a very clear objective function; i need this code path to run in this many microseconds/nanoseconds, so i can hit the scale numbers I need. Claude/Sol can extract prod logs and build a representative micro-benchmark, and then hill-climb on it autonomously. Thats straight up the fairway for the training set, even if it often has a high bar for finishing and requires sophisticated Workflows or lots of tokens to explore the search space.
E2A: sorry sorry, typo, I meant microGPT. I can see why this would be confusing.
So despite its importance much of it is actually pretty in-distribution.
Personally, I'm more on the LLM code is pretty bad from a design point of view, particularly when much of the code is already LLM generated.