> AI isn't good at software maintainability
We're a few years into a new technology that is still improving. This is a point-in-time critique.
We're a few years into a new technology that is still improving. This is a point-in-time critique.
My belief is that paying off tech debt requires a better model than creating it. At the same time there are people who will create tech debt no matter the tool.
Should the models stop improving, the debt will pile up.
Alternatively stated: codebases will expand to the limit of an organization's ability to manage them, so the equilibrium will remain at the point of near, but not total, incomprehensibility.
It was mostly 256k, then it went to 1M and now it has stalled there.
Meanwhile, the people who work on actual products that matter have a valid criticism that can’t be dismissed with “some humans don’t even manage to do that.”
Probably a vast majority of software written never actually gets interacted with by a user. A vast majority of VC-funded software probably never gets interacted with by a user.
Feels pretty great to work on a product that actually has a user, and that user isn’t actually the product (and that user isn’t being manipulated or exploited). Tiny, tiny minority of the software industry I work in.
If you don’t think about it as a per-software criteria, but per-feature criteria, the vast majority of software features developed are probably completely ignored or even untouched compared to the “critical few”.
Very, very long tail distribution. The most executed 100 lines of code (maybe some Java class init snippet, or some Linux kernel snippet) is probably executed more per day than the per-day bottom 80% of all code combined. Maybe the bottom 99.9% of all code combined.
[1] There is probably a more fitting word, I am just reusing incompetence here, but that is not a really fitting description, I think. I would maybe say carelessness or something like that, but a single word is not going to capture the issue accurately.
But the point OP was trying to make is that because LLM harnesses output code that is at least better than the worst, say, 20% of developers then we should be fine with it. Meanwhile, I’ve worked in shops where you have to be way above the worst 20% in order to keep your job or even be hired in the first place. And it isn’t some crazy lose-sleep-over-it, stressful requirement. It’s just that literally if you’re not good enough your work will be identified as a liability and you’ll be let go. It’s only happened to 3 people fwiw.
LLMs commit crap, and read the "pattern" back, and consider it as gospel and repeat it all across the code base...
For starters, LLM’s need to stop being our friends. But that won’t happen because the dopamine loop is baked in on purpose.
Generally though, you are also investing your time into leveling up junior engineers to take over responsibilities from you. I just never really see that happening with AI. Even as it gets "better" technically, there's no real growth pattern to its work and it doesn't understand ownership or responsibility.
But if scaling isn't a problem, then sure just write it yourself.
It doesn’t take much effort to setup cross-agent reviews and automatic reviews for slop and accretion, while directing design decision questions back to the human to consider. I have had a considerable increase in throughput of code that I designed and made the important decisions about, and that I’m pleased with the quality of, although as always in these discussions, someone will be a long shortly to tell me that that implies I must be a terrible engineer.
Mostly stalled now...
If the models are as capable in a years time as they are today you could say they have stalled.
And the observation that it's happening slower than it was means, not that it's stalled, but at least that it's not exponential anymore. The exponential phase is already over. That has implications for where we think the plateau is likely to be.