The ‘flawed five’ engineering productivity metrics
leaddev.com
leaddev.com
- Avg time from code review requested to code review picked up
- Avg time to complete code review
- Avg time from eng done to first customer using it
- Avg time from eng done to full production release
- Fraction of tasks started that never reach a customer
The time to complete a code review is there (for example) not to focus on the amount of time actually spent performing the code review but to focus on all the waiting around the actual code review, which is typically much much longer than the time spent doing the review itself. It's not uncommon to see organizations where an engineer will submit code for review and then have to wait a day or more for someone to pick up their request and then another day or more for that other person to get around to reviewing it. If there are comments on the commit that need to be responded to, you can see additional delays. These "minor inefficiencies" can have huge impacts on the poor dev who is trying to get their code merged, and cumulatively they result in significant increases in feature latency, the total calendar time required to ship a feature.
How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable.
In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to use it.
I would even say that canceling many started tasks is directly correlated with a quick cycle time, by Little's law.
The last one is definitely in a different bucket for me than the first four. For starters, all of these are there primarily to encourage conversation. That said, the first four can make a lot more sense to try to graph and track and optimize. The last one tends to be more purely about driving a conversation.
At the top level, if you're genuinely doing a ton of learning along the way to shipping the right feature to the customer, then arguably that value is ultimately reaching the customer. If on the other hand you can't decide what the goal is and you keep changing your mind (as is often the case), then you tend to end up with a lot of dev investment made in things that simply never ship.
It's also worth differentiating technical "spikes" from feature "experiments." In my vocabulary, spikes are things where you're internally assessing a question like "could we do this" and experiments are things where you're externally assessing "do customers want this/does this have the impact we want." If you have a lot of experiments that don't reach customers, you're burning a lot of dev time on things that aren't actually experiments (because by this definition experiments need to reach the customer surface to deliver data). That's a signal you should probably be looking at. Spikes generally only reach the customer indirectly (through an eventual shipping feature), but if you have a lot of spikes that don't ever reach the customer in any way that's also a signal you should probably be looking at.
The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently.
A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress.
It's fundamentally not a synchronous process and I say a mistake to try to make it one.
Edit to address a sense I'm getting from several comments at once, that I may be a manager who doesn't understand or care about developers:
I AM the developer (& sysadmin) with a bunch of fresh and stale projects and it doesn't bother me at all. I always have work to do but it's not stressfull because that is about your boss(es) not about the number of open items. If anyone tried to say all that stuff needs to be done yesterday, sure that would be a problem. But all they really are is a spectrum of priorities. Some are just ideas little more than stubs for possible future interest, some would be nice but may never be justifiable in a strict easily quanifiable sense, you can't bill any customer for the hours, and yet would still be nice, would be one of the things that sets your product apart and attracts the customer in the first place, and some have normal priority, and when something high priority comes along, it simply displaces the rest. It just requires a boss who doesn't say that every single new item is always the most important.
That stack of available projects, and the age of half of them, doesn't bother me at all. Half the time, a different project is as good as a walk in the park for the reset / fresh look factor when stuck on something.
One perspective says that idle engineers cost money, so you should load up your engineers to 99 % utilisation with a huge backlog of tasks. The consequence of this is that tasks take, on average, months to complete because they sit idle most of the time, not being worked on.
The other perspective is that idle tasks is what's expensive. Thus you must keep your engineers lightly loaded (say 70 %) so they can, in your words, pounce on new tasks. With this setup, most tasks spend a negligible time waiting and most time actively worked on.
Now, how can idle tasks be more expensive than idle engineers? There are whole books dedicated to explaining this. I'd recommend starting with Reinertsen's Principles of Product Development Flow.
My pool of available work is not impacting whatever I happen to be working on at the moment. I am not doing 1 second of work on 60 different projects every minute.
Idle time is also where you do useful things like checking for updated dependencies, running a linter, generally reading up on the tech you're using, and making the code ready for future edits. Not so idle at all but not really the same as working on a ticket either.
- Time between idea first recorded and idea in production.
- Time between idea merged to main branch and idea in production (in some places this is negligible, in others where ideas spend most of their time hanging out -- done, but collecting dust).
- Time actively spent on task over the time between work started on idea and idea merged to main branch. (This is "flow efficiency" in lean terminology.)
Other than that I use the DORA metrics too. I measure defect rate by looking at proportion of merge requests that fix defects, because mostly that is a stable distribution.
If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers.
If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the system in exactly the way it's designed to be "gamed." In general, if devs are talking to each other and aware of what each other is doing, efficiencies are high and latencies are short. If no one is talking to each other, efficiencies are low and latencies are high. Talking to each other and coordinating with each other is a good thing.
Edit: granted, sometimes the result of said debugging is that it really is a hardware problem, and with how unstandardized people are, that's actually common enough that you'll likely run into it. But you don't fix (for example) a disk drive that reports writes as complete before they're peristed by stack-ranking drives and trying to pick the ones that lose data less; you fix it either by replacing the faulty drive immediately (assuming you can get working drives), or (more relevantly to dealing with people) by working around it in software.
Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.
The amount of active "eyeballs on code" time to complete a code review can of course vary tremendously. That said, when I see code reviews taking a week to complete, only very very very rarely is that a sign that developers are routinely spending a solid week of eyeballs-on-code time completing those code reviews. Most of the time it involves a relatively small amount of focused review time and a huge amount of not actually getting around to starting doing the code review time.
If everything in your shop is so well handled that you're not wasting any time waiting for code to get reviewed, in practice your shop is probably also already making good choices about how to structure code for efficient reviews as well (but that's more an anecdotal observation about correlations, not directly an observation about causality).
Every time I've made a larger PR I've regretted it and subsequently split it up into smaller ones. I get higher quality feedback and faster feedback to boot that way.
Or you just get a "ship it" without more than a cursory glance.
Someone once argued to me that each feature PR should have very few surprises if the feature has been planned and communicated well with affected parties before implementation even begins. The opposite is often true in some organizations.
I have also found that high review latency begets large code reviews. It's kind of a vicious feedback loop. If you always review PRs immediately, people are less afraid of splitting up their work, and PRs will become smaller. (It's hard to "stack" reviews, so people avoid it. That means that 3 PRs become 1 PR, so they can be mentally done with the task and start something brand new while waiting a week for your code review. The alternative is hoping that the bottom of your stack is approved without any requests to change anything, but what happens is that large changes are requested and the two floating PRs are now impossible to merge back in, or are irrelevant. If the bottom of the stack could be reviewed in an hour, then all that work wouldn't have been wasted, and it reinforces the good habit of doing 1 small thing at a time.)
I look at all of this as a systemic issue, and not a personal issue. The team needs to set an aggressive SLA, guided by what the engineer is expected to do while waiting for a review. Then engineers need to treat the objective seriously; code review is as important as any code you're writing. That's why you're part of a team, and aren't a 1 person army (which is a totally viable approach; I write a ton of code in my free time by myself, but I don't necessarily think that that's how I should behave at work.)
Personally, I have Github's Slack notifications on, and almost always start the review as soon as I get the notification. I know that the person waiting for the review has no work to do other than to build on top of this foundation; the less they do without feedback, the better. (Plus, they want to get their code Out There, that's why they wrote it.) I don't think your organization has to be that aggressive, but I'd aim closer to "hard interrupt" than "within 1-2 business days". (No, I don't review PRs that come in at 7PM on Friday night. Those wait until Monday morning.)
The other metrics listed a few comments above are really a function of organizational policies. Make your releases easy, so you do them often. Make sure that your PMs know what users want, so you don't implement code and throw it away. Make sure that CI is fast, so there isn't a delay in pushing to production. And make sure your process is realistic. My team used to spend a lot of time manually testing stuff in staging before releasing to production. It never found any issues that the test suite didn't find, and we prefer the automated tests anyway, so eventually we just made every commit to master a deployment to production. Click the "merge" button and it's out to the users. No unusual number of surprise production outages, just a vastly reduced time from idea to being in the hands of the customer. (Code reviews ended up being the limiting factor. People really don't like doing them!)
If you optimize this down, you punish any kind of innovative or ambitious task.
If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway.
And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough view to know if your software is such one.
Specifically, look at the "PM can't make up their mind" graphs (https://apenwarr.ca/log/20171213#slide13). That's the state that you're trying to avoid.
I think it's fine to have tasks like "investigate the feasibility of X" and finding it infeasible. To me, that outcome is similar to delivering a feature (your customers will pay less for that than an actual feature, though, probably). What you want to avoid is "do X at any cost", and then the first steps are "integrate with third party service Y" "get a support contract for service Y" "do a giant refactoring to better use service Y", and then finding that X is actually infeasible. Now you have complexity, cost, and wasted months without having anything to give to the user.
(Conflict of interest note: go ahead and do that. More work that never ships = more developers needed = higher pay for developers!)
And in the process of asking questions and poking around this person learns a lot about what's happening.
Its ridiculously dumb. I've heard those numbers matter less as you gain tenure and seniority. But for the new grads and lower level engineers, all the savvy ones were gaming these metrics. It sucked to work in that environment and was a big reason why I left.
The company care more about weeding out the really bad junior engineers than it does about rewarding the good ones.
Honestly I don't have a big issue with that because I've seen what happens when bad people stick around. It's never fun for the good people, though.
The software used would spit out individual metrics like "number of calls taken," and our supervisor used to look at that metric to make sure we were working and not slacking off at our desks, not on the phone.
People figured it out and would just pick up and hang up the phone ten times in a row to game the metric. In response, the supervisor started looking at "average call time" in addition to the number of calls, to make sure this wasn't happening.
So what people did instead would be pick up ONE call, and leave the line open for an hour, long after the customer had already hung up, in order to game that metric.
Seems like something similar could be done with these Amazon metrics.
And then on the even worse side, you have Amazon's stack ranking coming into play. Engineers can screw each other by blocking merges with needless comments, slow dripping reviews (eg problems that were in iteration 1 aren't brought up until iteration 3).
I won't deny that maybe I'm a paranoid person and that I read into patterns that weren't there sometimes. But the fact that those patterns could exist was enough to hurt my mental health to the point that I wanted to leave.
If anyone complains, I will take the extra time to turn it into one commit per character of code changed.
You want commits? I can give you commits. It's not what I'd really like to do, but by golly, I can do it. Commits by the thousands, by the millions (whoops, did I just import a library and run my code over it? sorry), anything you like.
I've never heard of such a thing. It's leadership principles only AFAIK.
The iterations per MR metric is one I know for a fact is used in PIP docs. Heard this directly from a friend that was PIPed.
OK so this is hearsay, your original post did not frame it as such, and you walk it back further here.
In addition other Amazon people seem to not agree, maybe don't post something like this next time.
What I would tell you that is verifiably true, is that myself and other engineers all worked under the impression that these metrics were being used. Whether or not they actually were I cannot say for sure, but I can definitely tell you that it affected the culture.
The OLR process at Amazon is notoriously opaque. The HR and management at Amazon are notoriously smart. Amazon has a bad working culture (obviously, this is in my opinion). I would say that it is my duty to share elements of it that were bad for my wellbeing, to help others make informed decisions about their own futures.
At the same time, metrics do have a place. Even flawed metrics, like the ones this article describes can provide value. When used together with qualitative evaluation and thoughtful analysis, it provides a more complete picture of what is going on in a team.
Metrics such as these provide an addition perspective on a team. A manager knows what their team should be doing. A good manager should have an intuitive feel of what is going on. A manager should have a good qualitative idea of how their team is doing. If the metrics do not align with the other perspectives, something may be off.
If a manager believes a person should be coding, the person is not bringing up any challenges, is reporting progress, and they produced 3 small commits over the past month, it is time for a conversation to find out more about what is going on.
Metrics are for asking questions, not answering them.Where these metrics would be useful is if the business actually looked at itself first when teams or team members underperform. In all honesty, I've never met a manager with this kind of mindset; they're usually captured by the belief of the above in some way.
Businesses do have an old way of determining whether a team is meeting its goals: KPIs. If the team is responsible for a succinct domain, problem, or stack then these KPIs are easy to draw and measure against because they reflect business outcomes rather than trying to normalize for how everyone on a team contributes.
Obviously this is a management problem and it’s not a single individuals fault. But the manager had like 90 contractors reporting to them and didn’t care that people had zero lines of code written.
The developer’s job was to code. I wanted to mention that as there are developers and designers who don’t code but are productive in other ways.
Having some metrics here can help sort this out before you call some senior dev at 12:30 in the morning for code they didn't even write.
It always amazes me how otherwise very smart people don't think through the consequences of "paid by the X".
I do not believe in the possibility of a "General Theory of Productivity," and management-by-numbers-alone is actively harmful, but I do believe in the possibility of measuring productivity in a useful way. Even "bad" metrics like commits per engineer per week can be useful at the right granularity, e.g., to do high-level velocity forecasting over a large, representative group of engineers during different times of year.
If you're wondering: different metrics are suited for different use cases, but as a baseline, I think the DORA metrics[1] are a reasonable starting point.
1. https://cloud.google.com/blog/products/devops-sre/using-the-...
IBM started measuring code defects vs working code in this way because productivity studies they did showed that fixes took much more time per LOC than new code and had other costs (customer sat, doc changes, reputation) besides.
Adding inline documentation would improve this metric.
This does explain some absolute dogshit products IBM made as maybe they were optimizing for this metric by having 1000 lines when one would do. I’m bitter from having to decompile and debug websphere in the 90s and 00s.
I think this runs into a problem is that programmers are good at minmaxing. So any rote metrics will end up being gamed pretty quickly.
On the "why they failed" aspect, 100 years is a pretty good run. There are lots of reasons IBM is less relevant today, but buggy software isn't one of them.
Excessive or poorly timed meetings, scope change, poor WFH distraction management, excessive support load, etc.
I'd like to think that we can assume people will be productive if we set them up for it.
Oldest MR - this should always be less than 2 weeks. This should normally be less than 1 week, but its not worth caring about at less than 2.
Unfinished sprints - sprints should finish with enough time left over to cope for an incident in the week. The extra time should be used for planning and continuous professional development (CPD). When someone is trapped in overflowing sprints it means they are deprioritising and undercompleting work that will come back to bite you.
Track other metrics for at most 3 months each, ideally only a month or a single week spot check. This prevents gaming and obsession whilst letting you reason about more nuanced behaviours.
I quickly figured out that under the new system, you could increase the amount of time you get for a particular patent application through a particular reclassification procedure called a C* (pronounced C-star) challenge. I'm surely not the only one who figured that out. The reason the C* challenge exists is to reclassify a patent so that it can be transferred to a more qualified examiner. But if it's not transferred then the amount of time you get can be changed. That's not necessarily nefarious as many applications have the wrong classification and would give you a lot less time than if they had the right classification. But examiners aren't incentivized to switch an application to the right classification. They're incentivized to change the classification so that the application gets transferred or change the classification so that they get more time. In the latter case I'd intentionally avoid adding (or even delete) any classifications that would reduce the amount of time I got. I don't suspect the long-term dynamics of this system are what USPTO management intends.
Impact is pretty clear to me. You can find the site of an impact by looking for the smoke coming out of the crater.
For example, this newer article claims “a manager said … ‘Pull request count is the new vanity metric’”, while the original article stated that “Unfortunately it's a vanity metric”. And the last thing this newer article says about “impact” is that the name “sends a strong signal about how it should be used, particularly by managers”, matching the original article’s statement that “‘Impact’ suggests to executives and managers how this metrics should be used.”
As the original article isn’t cited, this borders on plagiarism.
Julian's article was published May 2021, one month after my Leaddev article, so hopefully it's pretty self-evident that Julian plagiarized my article pretty blatantly. I don't know Julian but I've heard from my former customers that his company has copied a lot of the work I did for my previous company, Pull Panda: https://github.blog/2019-06-17-github-acquires-pull-panda/
It doesn’t seem true that Julian’s Haystack article (https://www.usehaystack.io/blog/software-development-metrics...) was published in May 2021, even though that’s the date at the top of that article. It was apparently originally published in November 2020, as seen in a snapshot from December 2020: https://web.archive.org/web/20201204170826/https://www.useha.... Your article does seem to have been published in April 2021, as you say: https://web.archive.org/web/20210419071103/https://leaddev.c....
Though your article was published later, the talk you gave on this topic (your YouTube link) was from December 2019, before either article. Julian Colina’s article quotes many phrases from that talk directly, so it does indeed seem that you, Abi Noda, are the original source.
If you set up an incentive system, people will (perhaps not even consciously) start trying to game that system. Maybe there's some way to use machine learning to turn performance metrics into a black box that considers every possible data point? (Now there's a terrifying idea.)
Metrics are useful to navigate BY, not to navigate TO. If you have skilled and experienced managers, you can get a lot of value out of all of the metrics listed in the article.
Now, I would contend that the rest of the productivity metrics depend on what the team and its parent structures are trying to achieve, and should be DIFFERENT over teams and over time. Maybe a team with a spaghetti-like legacy project will want to track LOC or cyclomatic complexity for a while. Maybe some other team will want to track the amount of transitive dependencies. Maybe some will want to optimize for onboarding (time-to-10th-PR or something like this). 1 size will not fit all, and the team and its management must work to figure out what success looks like w.r.t productivity based on the context at hand.
These kinds of metrics can involve people (e.g. add a feature to increase customer satisfaction by X points), performance (e.g. optimize a path to increases throughput by Y%), processes (e.g. fix a bug so security continues to match commitment Z), etc.
I've said it before and I'll say it again: finding ways to characterize people as bad at their jobs is about keeping salaries down, whether by accident or on purpose. Because those are metrics your boss's skip level manager looks at. And they 'work' until they don't, by which point the manager can move up or out and get a reset on the numbers being used against them.
I have yet to see any meaningful increase in a team’s productivity after they start tracking “developer productivity”.
Each time it results in a blow to developer morale and a pretty dashboard that management uses to retroactively justify their decisions.
If you want some form of rational decision when you develop a team, how else but with metrics do you make decisions?
It's a terrible answer, in no small part because it's true.
How do you manage people when you're bad at managing people and can't/won't look at it? You manage safe, crisp, numbers instead.
The best physics professor I ever had assigned a fuckload of homework every single day of class and graded it all himself. He probably worked 65+ hours a week, minimum. But you got feedback on all homework and that gave you an understanding of how well you were grasping concepts and on what you needed to work.
How can a manager do code review when he doesn't even have the capacity to know the details of the tasks?
Now imagine who you want to pay to do stuff and how to measure them. The best way is to see their actual performance unfold before your eyes and use your own intuition. See who gets things done and who doesn't. Who needs more handholding and who can work more independently. Some workers are much better "deals" than others from a company perspective.
You're completely failing to answer the question.
* Cost reduction; usually by a very arbitrary amount, despite you having no control over what's needed
* Uptimes; last job told me that I had to get 99.998% uptime, Googles global load balancer is only 99.9%
Externally to customers we had no promises of availability.
(For reference, I am talking about cases where a team of 5 devs takes 2-3 months to deliver a feature that would take a single independent developer maybe 2-3 days.)
I always encourage fellow engineers to vastly overestimate tickets. That‘s also important to set a comfortable pace with the business people who have zero clue how hard our work really is and prevent them from making us work hard.
Note that this goes both ways; the'll both wildly overestimate and wildly underestimate how hard various tasks are, often in the same conversation.
You mean snail or like moving in an "S"?
#commits are useful as a binary metric that a developer is alive, but trying to say one is more productive than another because they had more commits is pure madness that permeates an org so that I would detect it during an interview and avoid.
I don't think anyone is saying you should only look at the metrics and not the qualitative factors (what's in all those frequent commits?) but they definitely help drive conversations and decisions. The alternative (pure qualitative & gut feeling) is much harder to get consistent across an entire engineering department.
‘I don’t hear hammering! There should be more hammering!’
‘Lines of code’ is a useful metric for ‘likely ongoing maintenance cost’. ‘Impact’ is a good proxy measurement for ‘likelihood the change introduced a bug’.
If you encourage teams to increase those numbers you will get what you deserve.
IMHO, developer productivity is best judged by the humans they work with.
There is a critical error in this statement that some would label as 'subtle' but it's about as subtle as a rusty axe to the forehead. Judging people by lines of code doesn't 'incentivize [] practices that are counter to building good software'. It punishes you for writing good software.
While we do need to be aware of the negative consequences of inaction, that's a problem to be solved once you have stopped actively digging a hole. That's a problem to be solved once you have stopped pushing people into that hole. That's a problem to be solved once you've stopped publicly congratulating people for not being pushed into the hole. By you. With an audience. There is nothing remotely subtle or nuanced about that distinction.
There's an old adage that if you make laws people can't respect, then they will stop respecting the law. This is but one route to that problem.
But they are stupid to measure across teams or to compare teams or productivity. Velocity points are just an estimating tool, not a measure of value. It’s useful to know that a team usually produces 10 points per sprint but this sprint is 5 or 20. It just lets you know if your team is producing “normal” or not.
It’s useless to try to calculate that out of 20 teams the average velocity points are 10 per sprint.