Show me your data.
The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.
The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566
https://arxiv.org/abs/2302.06590
The METR study that you're likely talking about had a lot of nuances that don't get talked about, not to mention outright concerns e.g. this one participant revealed he had a pretty damning selection bias:
Lots of people we're now conversing with could be junior or mid-level, might have tried it for little prototypes/experiments, or for more trivial software like commissioned websites, and so on. They could all be benefiting from agentic coding workflows in ways that we don't. With the caveat that the study you talked about also showed even the experience devs felt more productive, so clearly the use of AI biases your perception of delivery speed.
The large array of context I suspect is responsible for some of that dissonance on online discourse.
https://scholar.google.com/scholar?hl=en&q=llm%20developer%2...
> The only study I’ve seen so far on LLMs and productivity
Why are you guys all citing the same study in this thread? There are several studies including a survey paper.
Or you've got some that takes like beginner devs that never worked a job and have them run through some unrealistically simple task.
Or a handful that are claiming productivity boosts, but they count things like faster than Google search, or fewer keystrokes.
Or when they measure the "code quality" they just mean the tests passes.
So when that study came out and I read the methodology, I was like, oh, this is interesting. And then the results were also quite surprising, and surprisingly the one with the best methodology had the opposite finding of all the crappy ones I've seen before.
Food for thought.
[0] https://williamcotton.com/articles/introducing-web-pipe
[1] https://github.com/williamcotton/williamcotton.com/blob/mast...
[2] https://github.com/williamcotton/webpipe-rs/commits/main/?af...