How I measure performance is how many features I can implement in a given period of time.
It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.
Someone already operating at the very limit of their abilities doing stuff that is for them high complexity, high cognitive load, detail intense, and tactically non-obvious? Even a machine that just handed you the perfect code can't 20x your real output, even if it gave you the source file at 20x your native sophistication you wouldn't be able to build and deploy it, let alone make changes to it.
But even if it's the last 5-20% after you're already operating at your very limit and trying to hit your limit every single day is massive, it makes a bunch of stuff on the bubble go from "not realistic" to "we did that".
A key skill is to sense when the AI is starting to guess for solutions (no different to human devs) and then either lean into another AI or reset context and start over.
I'm finding the code quality increase greatly with the addition of the text 'and please follow best practices because will be pen tested on this!' and wow.. it takes it much more seriously.
Most of the coding needed to give people CRUD interfaces to resources is all about copy / pasting and integrating tools together.
Sort of like the old days when we were patching all those copy/paste's from StackOverflow.
Too little of full stack application writing is truly unique.
It would be interesting to set up a MCP style interface, but even me copy/pasting between windows was constructive.
The time this worked best was when I was building a security model for an API that had to be flexible and follow best practices. It was interesting seeing ChatGPT compare and contrast against major API vendors, and Claude Code asking the detailed implementation questions.
The final output was a pragmatic middle-ground between simplistic and way too complex.
Also I disagree. For web dev atleast, most people are just rewriting the same stuff in a different order. Even though the entire project might be complex from a high level perspective, when you dive into the components or even just a single route it ain't "high complexity" at all and since I believe most jobs are in web / app dev which just recycles the same code over and over again that's why there's a lot of people claiming huge boosts to productivity.
The difficult part is reading thousand lines of unfamiliar code to measure the impact of a fix, finding the fix by reasoning about the whole moduke, designing a feature for long term maintainability,…
Note that all of them requires thinking and not much coding. Coding is easy, especially when you’ve done all the (correct?) thinking beforehand.
How much of the code you write is actually like this? I work in the domain of data modeling, for me once the math is worked out majority of the code is "trivial". The kind of code you are talking about is maybe 20% of my time. Honestly, also the most enjoyable 20%. I will be very happy if that is all I would work on while rest of it done by AI.
When you zoom in, even this kind of work isn't uniform - a lot of it is still shaving yaks, boring chores, and tasks that are hard dependencies for the work that is truly cognitively demanding, but themselves are easy(ish) annoyances. It's those subtasks - and the extra burden of mentally keeping track of them - that sets the limit of what even the most skilled, productive engineer can do. Offloading some of that to AI lets one free some mental capacity for work that actually benefits from that.
> Even a machine that just handed you the perfect code can't 20x your real output, even if it gave you the source file at 20x your native sophistication you wouldn't be able to build and deploy it, let alone make changes to it.
Not true if you use it right.
You're probably following the "grug developer" philosophy, as it's popular these days (as well as "but think of the juniors!", which is the perceived ideal in the current zeitgeist). By design, this turns coding into boring, low-cognitive-load work. Reviewing such code is, thus, easier (and less demoralizing) than writing it.
20x is probably a bit much across the board, but for the technical part, I can believe it - there's too much unavoidable but trivial bullshit involved in software these days (build scripts, Dockerfies, IaaS). Preventing deep context switching on those is a big time saver.
Yeah, I'm not a dev but I can see why this is true, because it's also the argument I use in my job as an academic. Some people say "but your work is intellectually complex, how can you trust LLMs to do research, etc.?", which of course, I don't. But 80% of the job is not actually incrementally complex, it's routine stuff. These days I'm writing the final report of a project and half of the text is being generated by Gemini, when I write the data management plan (which is even more useless) probably 90% will be generated by Gemini. This frees a lot of time that I can devote to the actual research. And the same when I use it to polish a grant proposal, generate me some code for a chart in a paper, reformat a LaTeX table, brainstorm some initial ideas, come up with an exercise for an exam, etc.
Tons of dev work is not exciting, I have already launched a solo dev startup that was acquired, and the 'fun' part of that coding was minimal. Too much was the scaffolding, CRUD endpoints, web forms, build scripts, endpoint documentation, and the true innovative stuff was such a small part of the whole project. Of the 14 months of work, only 1 month was truly innovative.
Maybe, but I don't feel (of course, I could be wrong) that doing boring tasks take away any mental capacity; they feel more like fidgeting while I think. If a tool could do the boring things it may free my time to do other boring work that allows me to think - like doing the dishes - provided I don't have to carefully review the code.
Another issue (that I asked about yesterday [1]) is that seemingly boring tasks may end up being more subtle once you start coding them, and while I don't care too much about the quality of the code in the early iterations of the project, I have to be able to trust that whatever does the coding for me will come back and report any difficulties I hadn't anticipated.
> Reviewing such code is, thus, easier (and less demoralizing) than writing it.
That might well be true, but since writing it doesn't cost me much to begin with, the benefit might not be large. Don't get me wrong, I would still take it, but only if I could fully trust the agent to tell me what subtleties it encountered.
> there's too much unavoidable but trivial bullshit involved in software these days (build scripts, Dockerfies, IaaS). Preventing deep context switching on those is a big time saver.
If work is truly trivial, I'd like it to be automated by something that I can trust to do trivial work well and/or tell me when things aren't as trivial and I should pay attention to some detail I overlooked.
We can generally trust machines to either work reliably or fail with some clear indication. People might not be fully reliable, but we can generally trust them to report back with important questions they have or information they've learnt while doing the job. From the reports I've seen about using coding agents, they work like neither. You can neither trust them to succeed or fail reliably, nor can you trust them to come back with pertinent questions or information. Without either kind of trust, I don't think that "offloading" work to them would truly feel like offloading. I'm sure some people can work with that, but I think I'll wait until I can trust the agents.
When I said that after you've done all the other stuff, I was including cutting all the ridiculous bullshit that's been foisted on an entire generation of hackers to buy yachts for Bezos and shit.
I build clean libraries from source with correct `pkg-info` and then anything will build against it. I have well-maintained Debian and NixOS configurations that run on non-virtualized hardware. I use an `emacs` configuration that is built-to-specifications, and best-in-class open builds for other important editors.
I don't even know why someone would want a model spewing more of that garbage onto the road in front of them until you're running a tight, optimized stack to begin with, then the model emulates to some degree the things it sees, and they're also good.
Lagged-ass electron apps are a choice: run neovim or emacs or zed, I have Cursor installed, once in a while I need vscode for something, but how often is someone dictating my editor?
I have to target OCI container platforms for work sometimes, that's what Arion and nix2container are for. Ditto package managers: uv and bun exist and can interact with legacy requirements.txt and package.json in most cases.
Anything from a Helm chart to the configuration for ddagent can be written from nixlang and into a .deb.
My current job has a ton of Docker on GCE running TypeScript, I have to emit compatible code and configuration, but no one stands over my shoulders to make sure I'm doing the Cloud Approved jank path or having a bash script or Haskell program print it. I have a Jank Stack Compatibility Layer that builds all that nonsense.
Job after job there's a little setup cost and people look at me funny, 6 months in my desk is an island of high-velocity sanity people are starting to use because I carry a "glory days FAANG" toolkit around and compile reasonable plain text into whatever ripoff cloud garbage is getting pimped this week.
It's a pretty extreme workplace where you can't run reasonable Unix on your own machine and submit compiler output instead of typing for the truly mandatory jank integration points.
I am wondering, what sort of tasks are you seeing these x20 boost?
I scoped out a body of work and even with the AI assisting on building cards and feature documentation, it came to about 2 to 4 weeks to implement.
It was done in 2 days.
The key I've found with working as fast as possible is to have planning sessions with Claude Code and make it challenge you and ask tons of questions. Then get it to break the work into 'cards' (think Jira, but they are just .md files in your repo) and then maintain a todo.md and done.md file pair that sorts and organizes work flow.
Then start a new context, tell it to review todo.md and pick up next task, and burn through it, when done, commit and update todo.md and done.md, /compact and you're off on the next.
It's more than AI hinting at what to do, it's a whole new way of working with rigor and structure around it. Then you just focus fire on the next card, and the next, and if you ever think up new features, then card it up and put it in the work queue.
If one of these things isn’t true, you’re either a fool or those productivity increases aren’t real.
A simple example: if someone patents a machine that makes canned tuna 10 times faster than how they're currently being made, would tuna factories make 10 times more money? The answer is obviously no. Actually, they'd make the same money as before, or even less than that. Only the one who makes such a machine (and the consumers of tuna cans) would be benefited.
10x to 20x is in relation to time, so something that would have taken 2 weeks (80 hours) would be done in 8 hours to be 10x.
> Do you clock off at Monday lunchtime and spend the rest of the week playing video games? Did your boss fire nineteen developers and give their jobs to you?
In other words, how are you taking advantage of all that extra time you claim to have?
Claude code has made bootstrapping a new project, searching for API docs, troubleshooting, summarizing code, finding a GitHub project, building unit tests, refactoring, etc easily 20x faster.
It’s the context switching that is EXTREMELY expensive for a person, but costless for the LLM. I can focus on strategy (planning features) instead of being bogged down in lots of tactics (code warnings, syntax errors).
Claude Code is amazing, but the 20x gains aren’t evenly distributed. There are some projects that are too specialized (obscure languages, repos larger than the LLM’s context window, concepts that aren’t directly applicable to any codebase in their training corpus, etc). But for those of us using common languages and commodity projects, it’s a massive force multiplier.
I built my second iOS app (Swift) in about 3 days x 8 hours of vibe coding. A vocab practice app with adjustable learning profile, 3 different testing mechanisms, gamification (awards, badges), iOS notifications, text to speech, etc. My first iOS app was smaller, mostly a fork of another app, and took me 4 weeks of long days. 20x speed up with Claude Code is realistic.
And it saves even more time when researching + planning which features to add.
There should be a FOSS project explosion if those numbers were true by now. Commercial products too.
Jokes aside, if 20x was on the table for any kind of meaningful work we wouldn't need to wait for much of anything, entire parts of industry would be invented and technically reworked by now. It's most likely ~1.25x for what is mostly trivial work that approaches 95% boilerplate and zero actual design work.
If you read calrain's posts the 20x number is taken from the "fact" that sometimes (not consistently or most of the time) something that was estimated at 2 weeks or 80 hours (who knows what it was and how that number came to be?) took 2 hours instead. That's not just some minor detail; it's just not a sound way of thinking about productivity increases.
Words without actions are junk. You are asserting something you have no proof for. Proove it then. Amaze us all with your productivity, out in the open. Shred those pilled up open issues on open source projects and then give us a report of how fast-easy it.
If it is "easily true" you'll be done by next month
But I agree, you don't need to prove anything to me. You need to proove it to yourself first.
Stage magicians say that the magic is done in the audiences memory after the trick is done. It's the effect of the activity.
AI coding tools makes developers happier and able to spend more brain power on actually difficult things. But overall perhaps the amount of work isn't in orders of magnitudes it just feels like it.
Waze the navigation app routes you in non standard routes so that you are not stuck in traffic, so it feels fast that you are making progress. But the time taken may be longer and the distance travelled may be further!
Being in stuck traffic and not moving even for a little bit makes you feel that time has stopped, it's boring and frustrating. Now developers need never be stuck. Their roads will be clear, but they may take longer routes.
We get little boosts of dopamine using AI tools to do stuff. Perhaps we used these signals as indicators of productivity "Ahh that days work felt good, I did a lot"
You're not "stuck in traffic", you are the traffic. If the app distributes users around and this makes it so they don't end up in traffic jams, it's effectively preventing traffic jams from forming
I liked your washing machine vs. sink example that I see you just edited out. The machine may do it slower and less efficiently than you'd do in the sink, but the machine runs in parallel, freeing you to do something else. So is with good use of LLMs.
For Waze, even if you are traffic and others go around you, you still may get there quicker and your car use less energy than taking the suggested route that feels faster. Others may feel happier and feel like they were faster though. Indeed they were faster but might have taken a longer journey.
Also, generally most people don't use the app around here to effect significant road use changes. But if they did im not sure (but I'm having fun trying to think) what metaphor we can apply to the current topic :)
Can't help but note that in 99% cases this "difficult things" trope makes little sense. In most jobs, the freed time is either spent on other stupid tasks or is lost due to org inefficiencies, or is just procrastinated.
Please don't speak for all developers when you say stuff like this
AI coding tools make me miserable to use
Like: Why isn’t this working? Here Claude read this like 90 page PDF and tell me where I went wrong interfacing with this SDK.
Ohh I accidentally passed async_context_background_threading_safe instead of async_context_thread_safe_poll and it’s so now it’s panicking. Wow that would have taken me forever.
So I work 8 hours a day (to get money to eat) and code another 4 hours at home at night.
Weekends are both 10 hour days, and then rinse / repeat.
Unfortunately some projects are just hard to do and until now, they were too hard to attempt to solve solo. But with AI assistance, I am literally moving mountains.
The project may still be a failure but at least it will fail faster, no different to the pre-AI days.
It means you can replace a whole team of developers alone.
I can believe that some tasks are speed up by 10x or even 20x, but I find very hard to believe it's the average of your productivity (maintaining good code quality)
(I don't think it's 20x, it's most likely hyperbole. People aren't that unique and it's not hard to see that people who use LLMs are often lulled into thinking they're more valuable to them than they actually are, especially when they "do more", i.e. they're a magic little person program that seems to do tasks on their own as opposed to glorified auto-complete that probably by raw numbers is actually more productive.)
So me finishing a carded up block of work that is expected to take 2 weeks (80 hours) and I get it done in 1 day (8 hours) then that would be a 10x boost.
There are always tar pits of time where you are no better off with AI, but sometimes it's 20x.
I've setup development teams in the past, and have have been coding since the late 70's, so I am sort of aware of my capabilities.
It super depends on the type of work you're doing.
I mean, it's literally unbelievable.
This is absurd measuring. You can’t in good faith claim a 20x improvement if it only happens “sometimes” and other times it’s a time sink.
The more detail you keep providing in this thread, the clearer it becomes your assessment lands somewhere between the disingenuous and the delusional.
Does that mean you deliver the same amount of code in the same time with 20x less bugs?
Or the same quality code in 20x less time?
Or 10x less bugs in 2x less time?
If you had a hammer which could drive a nail through a plank 20x faster but took 60x longer to prepare before each strike, claiming 20x gains would be disingenuous.
Sometimes it doesn't help at all. Other times it spits out several hours of work in seconds.
It's like asking what is the weighted average of 1 and infinity? Even if you can quantify how many 1s and how many infinities there are, the answer is always going to be nonsensical.
I ended up asking it how it wanted to work and would an 'AdminKit Template' work to get things moving.
It recommended AdminKit and that was a good move.
For me, custom UI's aren't a big part of the solution, I just need web pages to manage CRUD endpoints to manage the product.
AdminKit has been a good fit so far, but it was a fresh start, no migration.
It mentioned AdminKit and it worked out pretty well.
Recently, there was story about developer who was able to crush interview and got parallel full-time jobs in several start-ups. Initially he was able to deliver but then not so much.
Somehow your case is reminding this to me, where AI is this overemployed developer.
You can build a new product company with 20 people. Probably in the same domain as you are in right now.
When a measure becomes a target, it ceases to be a good measure.
So were the people taking the study. Which is why we do these, to understand where our understanding of ourselves is lacking.
Maybe you are special and do get extra gains. Or maybe you are as wrong about yourself as everyone else and are overestimating the gains you think you have.
https://repo.autonoma.ca/notanexus.git
I don't know the PDF.js library. Writing both the client- and server-side for a PDF annotation editor would have taken 60 hours, maybe more. Instead, a combination Copilot, DeepSeek, Claude, and Gemini yielded a working prototype in under 6 hours:
https://repo.autonoma.ca/notanexus.git/tree/HEAD/src/js
I wrote maybe 3 lines of JavaScript, the rest was all prompted.
How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study.
> Instead, a combination Copilot, DeepSeek, Claude, and Gemini yielded a working prototype in under 6 hours
Six hours for a prototype using four LLMs? That is not impressive, it sounds insane and a tremendous mess that will take so long to dig out of the prototype stage it’ll effectively require a rewrite.
And why are you comparing an LLM prototype to a finished product “by hand” (I surely hope you’re not suggesting such a prototype would take sixty hours)? That is disingenuous and skewing the numbers.
I have over 20 years of web development experience and 40 years of general experience writing software. I wrote the authors and they confirmed my thoughts:
"I totally believe it! Per the paper abstract, we find many factors driving results - and one of the factors is how experienced the developers are on the codebase, and how big/complex the codebases are.
"Given that this was a new and unfamiliar domain and new codebase, I would expect there to be much more speedup than the domain we studied!"
> Six hours for a prototype using four LLMs?
They have limits on the number of queries, so I used four different LLMs in tandem to circumvent query limits. I didn't write it four times using four different LLMs.
> it sounds insane and a tremendous mess
I posted the code. It's well organized, has few (if any) encapsulation violations, sticks to OOP quite well, works, and---if I knew the PDF.js API---would be fairly easy to maintain.
Yes, I stand by my claim that writing this annotation editor (PHP, HTML, CSS, and JS) would take me about 60 hours by hand and about 6 hours using the LLMs.
I'm leaning into the future growth of AI capabilities to help me here, otherwise I'll have to do it myself.
That is a tomorrow problem, too much project structure/functionality to get right first.
With most projects where innovation is a key requirement, the goal isn't to write textbook quality code, it's to prove your ideas work and quickly evolve the project.
Once you have an idea of how it's going to work, you can then choose to start over from scratch or continue on and clean up all the bits you skipped over.
Right now I'm in the innovation cycle, and having AI able to pick up whole API path strategies and pivot them, is incredibly amazing.
How many times have you used large API's and seen clear hands of different developers and URI strategies, with an AI, you just pivot.
Code quality and pen tests are critical, but they can come later.
In my experience, no.
These kind of shortcuts taken at the beginning of the project is why velocity have a sharp descent after some times. Because you’re either spending time undoing all of it (unlikely to be allowed) or you’re fighting in the code jungle trying to get some feature out.