The AI Productivity Gap
bjorg.bjornroche.com
bjorg.bjornroche.com
AI compresses implementation time for an individual engineer, but architecture decisions, design reviews, integration, testing, deployment, and production validation remain largely serial activities. If code generation speeds up by 5x while those bottlenecks don't, you've mostly increased the team's work queue rather than its throughput.
With the current capabilities, models still need constant babysitting and course correction. An engineer who lacks the skills to guide them can end up creating more work for the rest of the team. AI makes it easy to generate code faster than you can understand it, and that cost is paid during code review, debugging, and maintenance by colleagues, whose confidence in that engineer's skills may be affected by his use of AI.
What looks like a productivity gain for one engineer can become a productivity loss for the team as a whole.
Amdahl’s law remains unbeaten
For anyone who hadn't heard this before, like me. <3
In computer architecture, Amdahl's law (or Amdahl's argument[1]) is a formula limiting the speedup of a task as resources are added to the system executing that task.
The law can be stated as[disputed – discuss]:
the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used.[2]
Amdahl's law is often used in parallel computing to predict the theoretical speedup when using multiple processors.
Started to write up a comment, got a bit wordy, turned into a blog post: https://senkorasic.com/articles/ai-amdahls-law
Programming has always felt like converting thought-stuff into something that the computer can understand. That's still the same, it's just a different language, and much faster.
My hot take is: if people are saying AI code is bad, yet they're using cutting-edge models like Opus 5, then those same people are writing bad code even without AI.
I'm finding that AI today can write excellent code, as long as you plan, review, and help it along with a little guidance.
Integration, testing, deployment and production validation activities are also, depending on you setup, available for AI to work on. It depends on the type of stack that you have and how your hosting is set up but with an AI-friendly set up there's a lot of time to be saved here too.
You'd be opening yourself up to even more babysitting. Would that really save time, or improve quality? I'm not convinced, to put it mildly.
also, as far as quality goes, once something is automated that potential breakage puts a cap on how much improvement/quality you can achieve later... i'd guess llms are a bit more malleable there, but idk, i still see this issue even with skills and such...
Noneetheless, if the blog article already determines a gain of 15% and you have to factor in the cost of tokens, the company should still reduce the team size by 10-20% or by 1-2 people per 5-10 person team.
Which is still a massive issues for everyone who writes code.
Btw. when i was writing code full time (not being an architect), i for sure wrote like 90% of my time code.
I'm doing an experiment though having a code review agent becoming better and better doing code reviews how i would do it. Which leads also to a skill which fixes issues before the code review agent even has to find them.
We can now automate cases which were unthinkable. We can even keep the code review quality if i leave (Its not perfect yet but its better than what we had before)
Even with the same amount of code, AI code is less trustworthy* and requires more attention... but we know it won't be the same amount, it will be more. This means it will take longer to review, or there will be unforeseen consequences of not spending that extra time.
*meaning no human eyes have looked at it and said "this doesn't make sense", or "this is cheating", or "this doesn't meet requirements", and won't be caught until code review if at all.
If it's a general signal that's easy for you to recognize at a glance, it's a signal that's natural and easy for an LLM to replicate.
They're much worse at making the underlying structure work. Not incapable at all, especially not the modern LLMs. Frontier models kick ass. But it's true that an LLM denies you a lot of the classic "tell at a glance" by its very nature.
which look nice on a surface level but obfuscates real understanding of the code and the actual data structures being used. Your end result is pretty and reads nice, but is bloated and difficult to reason with code.
Sounds like Uncle Bob disciples
What a nightmare, AI only knows how to write crappy Clean Code*
When reviewing human code I focus on specific parts because I know that there are parts where a person will just not make a bug (unless very junior).
AI on the other hand, will not do an off-by-one mistake, but it will happily just delete perfectly working code for no obvious reason. Or monkey patch a dependency because it missed a config flag. Or generally fail in a very novel and creative way.
The effort it takes to review AI code is much greater. And this is in a code base I am deeply familiar with.
Imo the future lies in a solid core programs with powerful plugin frameworks that expect all plugins to be code that was never read.
What makes you think that?
Historically plugins have been kind of crappy because they were constantly breaking with updates. However if they only live as a spec, and are regenerated when needed they can easily survive API changes.
Bonus feature is that if all “installed” plugins are generated together, the llm can also find ways to avoid them being buggy due to weird interactions.
All this while keeping the main program from crashing.
Security wise the spec can also be inspected using a trusted LLM. It is trivial to hide exfiltration or malicious code in plugin/extension code (e.g.: honey). But it is much harder to hide it in a spec.
mind you most of the stuff posted here is in regards to big tech - even though it's 'hacker' news.
If you think that, you aren't reviewing human code closely enough IMO. Human code and AI code should be scrutinized equally closely. Or rather, if you're relying on where you think the code came from rather than actually, y'know, reviewing the code itself, then you aren't doing a good job.
That whole experience of going deep for a while into LLM coding, then trying to leave it behind made me pretty pessimistic about the future of our profession. We are creating a whole industry of people delegating their ability to work to a software stack currently controlled by basically 2 companies (that both have very sketchy financials). Doesn’t feel healthy
But lets be real, anything moderately complex that is out of the domain of publicly available sample code is hit or miss compared to the time invested running the loop. I'd much rather invest the time in myself.
What a lot of people don't talk about is the inherent security nightmare of trusting ai agents and the sheer data exfiltration happening behind the scenes.
But even with that result I don’t think it’s something we should bet the whole industry on, and something I personally don’t feel comfortable relying upon
OTOH, we run extensive harness optimization, where everything is specified in advance, then a plan is made, then a naive review of the plan vs the specification vs the blast radius, then implementation, tests, then a naive review of test coverage, a naive review of the code vis a vis our code guidelines, a review for smells, a review for silo violation and architecture compliance, a reconciliation of the documentation, then planning the next subfeature, etc. probably 10 percent code generation, 40 percent documentation and planning and adversarial review, 50 percent automated adversarial code review.
We use a Claude for planning and generation, sol for adversarial review. Our metrics say we are about 2x. Productive, at a cost of about $300 per dev per month.
We are also shipping less bugs and better, more clearly written documentation (we use technical writing English style guides implemented by Claude)
I think a big part is the constant adversarial review by a different model with no prior context except the coding standards. Also important is context management, we do an onboarding and wrap-up for each session where we have a batch of continuity documents- Learnings, musings, and roadtrips where we let the most successful high-context sessions research and then write about something that they “ became curious about” during the session. That actually brings in a lot of insight to the team and occasionally is brilliant.
Also critical is crossing compaction barriers (standard re-onboarding protocols, writing transition documents prior to compaction, etc)
Also critical is being able to smell when a session is going off the rails. What we do there is sideline the session, wait for master to advance a bit, then have it do a “4c’s” review of the committed work, blast radius analysis, and remediation. If there’s still useful context left we put it into a project unrelated to its failure context. That salvages the valuable session context without staying in the failure trench.
It’s a lot like herding cats.
Do you mind outlining your stack around this. I know you mentioned python to support your verification harnesses but I am more interested in the agent setup. Are you specifically using Claude and it's skills with custom plugins or are you using other harnesses such as pi. I have settled on superpowers plugin across Claude and Codex, Cursor and most of my time is spent iterating through the design doc between Claude and Codex, implementing with either and starting another review cycle with the implementation, using TDD approach. It can be a lot of work but the end result is more than if I had done it myself. I am trying to formalize more. Anyway, thanks for the great insights.
Models are very much predictable these days (except anthropic models). The real issue stems from letting them work on their own for far too long. Also we are not controlled by 2 companies anymore as kimi k3, deepseek flash (and soon pro) as the ultra-cheap variants, glm 5.2 especially is a direct replacement for opus 4.8.
Models will only get better and cheaper I wouldn't feel too pessimistic and wouldn't feel too bad on relying on them to accelerate work and free up mental space from menial tasks.
As a personal side-note I never let my agents do architectual design I only use them for implementing. I always found the actual coding part of programming extremely boring and coming up with designs, experimenting and testing the fun part.
I find that mediocre programmers and LLMs are bad at both. They're helpful if you want to shit out some repetitive boilerplate or perform a complex search of some kind but otherwise you're better off without.
Also models baked into the silicon are able to achieve efficiency that is simply impossible to achieve with programmable circuits, there is a general slowdown in the raw capabilities that transformers can achieve and agentic tool use is simply an amplifier that will reach a wall eventually. It wouldn't surprise me if we saw within 5 to 10 years accelerator cards that you're able to purchase and plug into via usb-c that are able to achieve thousands of tok/s as well as api costs going down to what we already see with subscriptions today.
There has been quite a lot of off-ramping going on where people feel satisfied with the performance they're getting out of the models and simply staying there instead of using SOTA.
> accelerator cards that you're able to purchase and plug into via usb-c that are able to achieve thousands of tok/s
if i'm a programmer for example, even being a couple months old is a huge annoyance because programming languages and frameworks are changing all the time...
We give agents tools, the ability to read a man page, the ability to use web search. Knowledge cut-off is far less important than it used to be.
> hiring someone who is a good coder, but has trouble reasoning about systems, has no patience for working through hard problems with others, and can’t break down vague requirements into tangible action items.
Why not hire the excellent developers for the highly-technical skills they bring, and match them with architects/product managers who are the ones who have the big picture? Am I crazy to think like this?
The really is that you could have experts in each area doing what they’re good, which means letting programmers actually program most of the time, and let business analysts figure out requirements, product owners decide features, designers decide UX and design, QA perform in-depth testing… sure , every programmer will have to manage some of this themselves to not get blocked the whole time waiting for someone to decide something, but that is NOT the same as just having programmers handle everything!
I think that if you only ever hire programmers who are also kind of people person, you definitely have to accept missing out on the antisocial but genius ones who are very likely the only ones capable of tackling the really hard problems! Unpopular view, I know, but it takes a certain type of person to achieve excellence in some areas. Just look at the most successful artists, writers, actors and especially CEOs. Programmers are clearly in that category. I’ve seen “normies” trying to write a little code. They don’t last an hour before they decide it’s bullshit that you need a semicolon precisely placed for the code to not explode, or that they can’t compile on this system until you’ve installed some tool chain that requires a bunch of commands no one knows by heart but you just need to make sure to follow exactly, otherwise hell may break lose.
These are very convenient and vague enough excuses to single out whoever honestly says your architecture is stupid.
They lost me by begging the question in the very first sentence.
Since these numbers are made up, I may as well throw my personal anecdote in the ring. I find reading and reviewing far harder with coworkers who are using AI. Tickets contain about 5x as much meaningless junk as they used to, and testing notes - while far more thorough - are often now multiple pages in length. Reviews also contain much more code, people try to do more drive-by fixes because the models can generate those fixes so quickly, and people understand the code they're submitting far less clearly because the model is able to generate fixes they simply couldn't previously.
I feel less productive than I was a year ago, and I don't see my team shipping more features than they were previously. But everyone reports that they're far more productive. I don't get it.
We use a Claude for planning and generation, sol for adversarial review. Our metrics say we are about 2x. Productive, at a cost of about $300 per dev per month. We are also shipping less bugs and better, more clearly written documentation (we use technical writing English style guides implemented by Claude)
I think a big part is the constant adversarial review by a different model with no prior context except the coding standards. Also important is context management, we do an onboarding and wrap-up for each session where we have a batch of continuity documents- Learnings, musings, and roadtrips where we let the most successful high-context sessions research and then write about something that they “ became curious about” during the session. That actually brings in a lot of insight to the team and occasionally is brilliant.
Also critical is crossing compaction barriers (standard re-onboarding protocols, writing transition documents prior to compaction, etc)
Also critical is being able to smell when a session is going off the rails. What we do there is sideline the session, wait for master to advance a bit, then have it do a “4c’s” review of the committed work, blast radius analysis, and remediation. If there’s still useful context left we put it into a project unrelated to its failure context. That salvages the valuable session context without staying in the failure trench.
It’s a lot like herding cats.
0.75 to 0.75? Rework from review is also much faster. Now you don't have to tell a peer to rework a bit here and there for obvious reasons and spend time on a new loop. The review process isn't atomic.
Our production pipeline is faster across our very large organization, after implementing AI processes.
> Tickets contain about 5x as much meaningless junk as they used to
This is a process problem. Developers should be able to answer questions about their PRs, or you reject it. It's not a daunting blanket issue.
"Make me a picture of a house".
> AI proceeds to draw a house.
"No that's not right, it should be a red bricked house. Not a brown one."
> AI redraws a red bricked house.
"No. It should have a 2 car garage, sit on top of a hill. Also it should have a front porch, and have a tree right in front."
> AI then draws a red bricked house on top a hill with a 2 car garage with a front porch and a tree right in front.
...
The issue is that people think AI should automagically create some vague idea in of theirs, without having to do the work of spelling out all the exact details. So AI (like people) must make some assumptions about what wasn't specified. Like since you didn't specify a "red bricked house" in your initial prompt, but merely a "house"... it had to come up with something as to the color, and did as you otherwise asked, but it didn't know you actually wanted a "red bricked house", since you never specified that detail. Hence why it failed to do what you wanted, and you had to "review and correct it".
Again this isn't solely an intelligence problem, but an inherent problem in language/communication, of unsaid assumptions/specifications. Of being unaware of what you don't know, unaware of your own assumptions, sometimes even being unaware of what you even want. It's why AI can't fully get rid of jobs in software.
But sometimes people just don't care. They just want a picture of a house made. Anything remotely resembling a house will do, not necessarily a solid one, or one that can withstand a magnitude 8 earthquake. You know... like something people can put up in minutes so they don't have to do hard work... such as a shabby old tent. And AI is very good at generalizing, so it can in fact achieve this. So everyone reports they are far more productive now, putting up tent after tent. Meanwhile, the people responsible for the slop have a nightmare to review...
Like so many senior developers I have encountered. That stuff is good for CV.
It's a very O-ring problem.
I'd even say the productivity gap is even smaller, if not negative in some areas...
There's no real way to get to a 10x developer nowadays. Even if a company somehow achieved the magic productivity increase in all employees you still need a 10x consumer to gulp it all down.
Empirical evidence through observation or self-reporting, sampling in some meaningful way, would obviously be preferable but is also often just not practical.
Guessing at numbers to check whether your thesis even works with some plausible assumptions is a meaningful first step and to my mind a good way to reason through something like this and make it discussable.
A possible outcome of such an exercise is also that for your thesis to work out you need to make wildly implausible assumptions, so that helps you to discard that thesis.
From my perspective this is a very useful way to approach a hypothesis where empirical evidence is scarce or at least hard to get. No reason to dismiss it immediately – especially since the fact that those are guesses was never hidden.
That's PoC-level, happy-path-only engineering and LLMs are very dangerous at inserting subtle hardcoded values and cheats along the way, which makes it really hard to spot them later on. You need a system to whack them before they infect your codebase terminally, be it hyper vigilance, more SCA, more TDD, SDD... Otherwise you'll wake up one day with a production incident at 3 am and wonder how that code was even allowed in the release bundle.
Beginners should follow the practice described in the article: have the LLM propose code, then type it manually. This forces you to examine each step, question unfamiliar decisions, and build a stronger mental model of the codebase.
Experienced developers who already understand the underlying concepts may find autocomplete more practical when writing from scratch. It preserves control over structure, comments, and coding style while preventing tools such as Opus or Fable 5 (Both of these models loves adding tons of comment in your codebase) from generating bloated code, excessive comments, or patterns that do not match the existing codebase.
It multiplies both good and bad decisions. Both mine and it's 'own'.
I can get some things done 10x faster and it might even catch mistakes or help me solve something difficult.
But if I am being lazy or complacent then it bites me that much harder.
Thought it might be an interesting read, however have up just after reading the first line.
For the context, code had always been a copy-paste exercise, big part of it was understanding and differentiating between the different choices. Along with it people were growing as engineering practitioner's too. Human learning still needs to happen if they are expected to fix the code when LLM gives up.
LLMs are quite useful tool in themselves, however the hype has unfortunately polarized the population.
Familiarity with what's already existing affects how much AI helps as a research tool.
A new developer to the team can research quickly using AI, while by contrast, experienced folks won't gain much vs. just using their current knowledge.
It shows up as a new team member coming up to speed impressively fast. But since it doesn't help the rest of the team it also doesn't contribute much to overall team productivity.
And I can do even better than that if I design the codebase specifically with LLM coding in mind, making choices that make it hard or impossible for the LLM to make certain categories of error it tends to make, and make it easier for the LLM to observe the results.
And it's test cases can sometimes leave a lot to be desired.
AI salesperson claims that AI works in principle.
the one whats unclear to me is, will there still be meetings.
One example, let's say there is a side bet that makes everyone 10x more productive with a success rate of 1%
It takes 2 hrs to make the bet wit agent orchestration.
10 people can get this done in their spare time freed up by AI in 5 weeks.
Bet cashes in and you are much faster at everything.
It won't feel faster. Because the brain probably scores emotionally in roadblocks cleared per hour.
Back when you got a single punch card loaded in a day it felt like a fucking win.
The other factor is you get paid the same and there is more disruption and competition and job insecurity.
But objectively value gets shipped faster using AI.
Just not much if you go the faster horses route with AI. You need the cars. (Or planes!)
We have had interns come in and do 5x more work than their predecessors using GenAI. Senior devs spend most of their time planning and reviewing now and Junior devs can implement. Both with the help of GenAI.