Is anyone actually seeing a shift towards improved structure rather than more code, faster? We seem to be living in the Gatling-gun version of the picture John Carmack drew when modernity also ushered into precision marksmanship...
Like one of my personal projects is a sort of "middleware" for Godot, and it needs to be lean, precise and match certain rules, because it's meant to be used by many games, so I can't blindly trust AI to generate code for it, it'd be too sloppy,
BUT Codex reviews have been a great help in finding bugs that would have taken me ages to even notice on my own. Some of those AI findings helped me overhaul entire subsystems, still manually coded.
I'm making full-stack applications at the moment.
Yes. At work we recently finished a complete rewrite of the platform. The old codebase got abandoned and two new codebases got stood up. Previous stack was Phoenix LiveView and the new one is Phoenix API + Vue /w TypeScript. Zero code shared between the two. We took the opportunity to re-architect a lot of the core functionality and get rid of the tech debt that had been hounding us and killing our velocity for the past six years.
We finished the rewrite in two months. A little over 700k LoC total. 95% of it was planned, orchestrated and written by AI. We could have gotten it done faster, but we were well aware of the potential for slop and thus paid an absurd amount of attention to (and tokens for) code quality. For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous, plus a ton of invariants enforced via deterministic checks. Everything from standard linting and formatting rules to AI-necessitated stuff like "comments cannot be more than five lines each" is in there. At the end of each week, we get together for ~30 minutes to review each bug that was fixed that week (there haven't been that many) and try to figure out if more deterministic checks can be added (or new ones extended) to prevent that class of bug from re-appearing. We actually have an Architecture epic with a dedicated Fable agent orchestrating all the initiatives and tickets in it and keeping track of improvements (and the occasional regression). So things like accessibility, usage of shared helpers, enforcement of common patterns, etc. are all kept track of.
The difference between the old codebase and the two new ones has been night and day. The new codebases, combined, are ~35% larger than what we had before, and yet there are way fewer bugs (we actually got suspicious at first and had to check to make sure our logging instrumentation was configured correctly). Performance is also stellar, features have been a lot easier to add and extend, and everything is a lot easier to find. When someone asks a question or has an issue we have a pretty good idea of what part of the codebase to look in and we're usually right.
It wasn't all unicorns and rainbows. Not everyone on the team was proficient with AI-driven development, so they had to be brought up to speed quickly and taught the tools and also what works and doesn't. Code reviews were also difficult and time-consuming. We had to deal with our anxieties about working in two new codebases none of us had authored much code in ourselves. What made all of it worse was that we were racing towards a deadline imposed by external factors so we all ended up working a lot of hours. It was essentially between getting it done this summer or waiting until next summer and we bit the bullet and did it. It absolutely would not have been possible without AI.
look, your application works, right? so it doesn't really matter what you or i think, and this is why AI matters. but this, your "100% test coverage" - that is pure slop. just 20 years ago, all the most popular software shipped with NO tests. are you getting it?
not sure what you're point is here. It sounds similar to "we use to use blood letting and leeches and doctors didn't clean their hands and everything was fine so what are you getting at?"
Good tests have real benefits. The fact that people shipped without them in the past in no way suggests they aren't needed or have no point.
Not really, but I can see why some people think that.
We treat 100% test coverage as "required, but by itself not sufficient". It doesn't give us false confidence that everything will be perfect or anything like that. But it provides us with the discipline to make sure no corners are cut, and the bugs that are fixed don't come back.
One refreshing aspect was that during PR reviews we stopped debating whether something needed test coverage. Instead we focused on what was being tested and how.
I’d be curious to know what percentage of the time spent implementing tests would have otherwise gone to discussions about whether to implement them or not. ;)
Cars today are safer than ever. Drivers (in my memory anyway) have never been worse.
However, what actually happens is that careless drivers often cripple or kill innocent bystanders in other vehicles as a result of their poor driving. That's why seatbelt laws and improved vehicle safety features are a good thing.
Not really?
About 20 years ago, I was working on Firefox and we had millions of tests on CI. I was working on a host of other open source apps and they all had tests (most of them had no CI, of course).
Are you getting older? A lot of people anchor their intuition of time and history to a certain year. There are probably still lots of people who think the 1990s is not that long ago even though it’s now over a quarter century since it ended. Maybe you mentally default to 2012 or so, when it might be true that most popular software shipped without automated tests (although manual QA was a lot more extensive in 1992).
But 20 years ago is now 2006, and unit tests were well established as a best practice. Perl had extensive automated tests in the late 1990s that everyone who ever compiled Perl would have noticed, since they were run by default and produced obvious output. Kent Beck’s “Test Driven Development: By Example” was released in 2002, and popularized both the name and practice.
I agree with the spirit of what you wrote, but my recollection of the timeline is different. The first decade of the 2000s was peak Crazy Agile Advocacy, but IIRC it wasn’t until the 2010s that unit testing really became almost universal practice. Much before that and it was still tangled up with XP, TDD and lots of other things that certainly weren’t universally accepted as good practices (notwithstanding the strident advocacy of a certain group of consultants/authors/speakers/bloggers and their fans).
I remember, back in the mid-2000s, when we had some consultants brought in to talk about different aspects of quality and testing. There were several working groups, each led by one of those external consultants, and one of them was about unit testing. This was in a relatively large software development organisation for the time, a few thousand people, and while some parts of the organisation had some form of automated testing operating by then, it definitely was not the case that the well-known products produced by the organisation all had a unit test suite. Other practices we’d consider routine today, such as peer code reviews, were also in their infancy during that period: some were doing them, many were not, and generally we had much less experience of how to do them effectively than we have today.
As an industry, I don’t think we really matured in how even the most ardent fans of unit testing were writing test suites until the 2010s either. In the 2000s, we still had lots of people mocking the entire universe and then writing unit tests that were 99% testing those mocks because of 100% test coverage requirements, and similar dogmatic nonsense.
By the 2020s, I think there was much more awareness of that automated testing is generally a good idea, but there are different kinds/levels of automated testing and finding a mix that suits each project’s specific needs is important. One of the great benefits from the more recent AI tools, particularly the agentic ones over the past year or so, has been that it has clearly demonstrated both the value of a good automated test strategy and how much of a waste of time vacuous tests are.
2 months of 80 hour weeks is 4 months of regular weeks. Can you clarify how many hours your team was putting in? And how large was your team?
How?
I don't want to sound flippant, but if the point is to add human thought to the mix, that's a high review rate even when examining small tweaks to an existing, working product, even with substantial AI help to pre-filter major gotchas before you bother spending a lot of human effort on the review. That's only 20-30wpm, but a review isn't just scanning or reading code, especially if you're trying to figure out how a new system which doesn't run yet will fit together.
But then you need to watch for bugs coming from interaction with previous changes and in 700k loc that might be nontrivial. How do you know which states are reachable and which are not? That takes time.
It only takes a botched condition here (forgot a "not"? swapped "and"/"or"?), a swapped variable name there, code that looks ok, but isn't.
It's possible, I think, but humans cannot do repetitive mental work (like reviewing code) constantly without atrophy.
Can I review 130 SLoC/hour? Sure. Can I do it 80 hours a week? No. Not even a single 8-hour workday.
I could see a world potentially where they came up with a magic prompt allowing each proposed PR to be cohesive, shippable, well factored, and everything else you need to be able to actually review it at a higher level and be comfortable with the results, but I'm skeptical. That's a major innovation if they managed to do so even as a one-off, and that wasn't the thing they highlighted when talking about the project.
This is why my current view in terms of LLM productivity gains are:
- debugging and triaging 5x
- refactoring when you have tests in place 3x
- greenfield where you have zero tests is about 1.5x
We're mid-way through a similar process at work. Rewriting a legacy app in a new language, with new architecture and new features.
And it's a mess.
We're at 10x loc (admittedly, the new programming language is more verbose than the old one), comments make no sense. Yes, we have ~100% coverage, but most of the tests are meaningless. The agent keeps removing our tests to replace them with tests that are easier to pass, breaking code invariants, removing all the engineered data structures and replacing them with stringly-typed code, etc.
And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
AI made it possible, insofar as upper management would never have greenlit the project without AI, but I can't escape the feeling that we're building on quicksands.
https://www.hanselman.com/blog/stringly-typed-vs-strongly-ty...
Reviews? They were the first casualty.
If your code base is script-kiddy garbage, your models are only going to keep writing more garbage. You should throw it all away. I expect that isn’t an option. Try instead instructing the model that the code you have is legacy garbage written in a hurry that needs siginificant refactoring, but that right now I need one new feature written to these new standards instead. Take it one small piece at a time. Don’t let context go beyond 100k. Good luck!
Can you be more specific? "Improve structure" can mean different things to different people.
We've ensured that agents strictly adhere to code architecture rules, using both agentic review rules and deterministic CI gates. Everything from file naming, location and namespacing to the "shape" of each file that shares the same role are consistent. We can trace functions of the same role across the same set of modules, with abstractions and code reuse where they make sense. Once someone learns the code in one part of the codebase, their understanding of that "spine" translates to other areas - the only differences relate to the specific business rules governing those areas.
>> And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code. For example, if a PR author has made the choice to use soft-deletes when the product uses hard-deletes for everything else, that is flagged by the code atlas, and a human reviewer can use that to ask questions about it. Meanwhile, things like boilerplate are de-emphasized; any problems associated with them are reliably found and flagged by AI reviewers anyway. The refreshing part about this is that this has almost completely eliminated bike-shedding: people no longer argue about module naming or whether a comment is worded properly.
Yeah, we did that, too.
But every time I end up, for some reason, digging up deep in the code, I realize that it's not nearly sufficient in our case.
Also what kind of problems did you have with LiveView?