> ... just looking at LOC or PRs, which of course is nonsense.
That's basically a variation of "How can they prove anything when we don't even know how to measure developer productivity?" ;-)
And the answer is the same: robust statistical methods! For instance, amongst other things they compare the same developers over time doing regular day-job tasks with the same quality control processes (review etc.) in place, before and after being allowed to use AI. It's like an A/B test. Spreading across a large N and time duration accounts for a lot of the day-to-day variation.
Note that they do not claim to measure individual or team productivity, but they do find a large, statistically significant difference in the data. Worth reading the methodologies to assuage any doubts.
> A Stanford case study found that after accounting for buggy code that needed to be re-worked there may be no productivity uplift.
I'm not sure if we're talking about the same Stanford study, the one in the link above (100K engineers across 600+ companies) does account for "code churn" (ostensibly fixing AI bugs) and still find an overall productivity boost in the 5 - 30% range. This depends a LOT on the use-case (e.g. complex tasks on legacy COBOL codebases actually see negative impact.)
In any case, most of these studies seem to agree on a 15 - 30% boost.
Note these are mostly from the ~2024 timeframe using the models from then without today's agentic coding harness. I would bet the number is much higher these days. More recent reports from sources like DX find upto a 60% increase in throughput, though I haven't looked closely at this and have some doubts.
> Meta measured a 6-12% uplift in productivity from adopting agentic coding. Thats paltry.
Even assuming a lower-end of 6% lift, at Meta SWE salaries that is a LOT of savings.
However, I haven't come across anything from Meta yet, could you link a source?