What is developer productivity and how to measure it?
axolo.co
axolo.co
As a line manager, with software engineers reporting directly to you, you should be able to use your personal judgment to understand the productivity of your software engineers. Don't measure it with acronyms, with metrics like the number of commits, or by paying attention to how many hours a week people are working. Pay attention to whether people get things done, and are they getting big important things done, or only little nice-but-not-critical things. Make sure you communicate enough so that individual software engineers understand how you think and what you prioritize.
As a manager-of-managers, it is going to be very difficult for you to measure developer productivity. It's tempting to look at metrics like the number of code reviews a developer does. But these can at most be a sanity check, not the core metric to go for.
Instead, you can measure productivity of teams. Is the team getting things done, and are they big important things, or only little nice-but-not-critical things? Sometimes, a line manager will insist that everyone on their team is performing excellently, and yet you observe the team overall is not achieving very much. Probably one of the two of you is incorrect, and you should dig in to figure that out. The opposite also happens, where a manager states that everything is a disaster, but you observe that the team has actually delivered a lot.
The other thing you can do is to teach your line managers how to judge individual productivity. There's no silver bullet, it's just a natural outcome of having conversations about who is productive and who is not and how to tell and what to do about it, so be sure to have enough of those conversations.
None of this is easy to quantify, but the hard truth is, there is no natural mapping from numbers to developer productivity and it is usually a bad idea to try to quantify productivity. You are much better off using human language and intelligent thinking to evaluate productivity, rather than reductionist metrics.
Humans and relationships are nuanced, including work relationships and the responsibilities and expectations there. It's best to treat them as they are rather than trying to shoehorn those things into such a sweet little checkbox.
Frameworks are alright, but they need flexibility built in. They certainly shouldn't be treated as religiously as they are commonly.
> One of the most common myths — and potentially most threatening to developer happiness — is the notion that productivity is all about developer activity, things like lines of code or number of commits. More activity can appear for various reasons: working longer hours may signal developers having to "brute-force" work to overcome bad systems or poor planning to meet a predefined release schedule.
The SPACE framework is not about measuring quantitative data only. I feel the need to explain how certain metrics might be interesting, but rather to identify key issues or unexpected events during engineering sprints. Without data analysis, you would not be able to understand why there is a drop of productivity during certain periods, and usually, those drops were created by the management (too many meetings or lack of follow-up)
Without some quantifiable metric, how do these teams know if their products are getting better or worse? The discussion always goes to measuring developer happiness & developer productivity because we want with some degree of confidence to be improving or at least maintaining these metrics.
Or maybe to introduce a new endpoint in your API takes X amount of boilerplate lines, Y files, etc and you post mortem new endpoints after your change to ensure that number is dropping.
Talk to people, find out what their problems are, quantify the problem, measure.
Often what frustrates people is a latency, which is something you can measure and track. Other times it is an ugliness, surprising footgun, or lack of conceptual integrity - these are fundamentally human experiences, and subjective assessment is the only way.
I see many people wanting to take writing code into the liberal arts domain, but I am of the opinion that it may be more useful if we can overlap it with the engineering domain. IMHO the goal should be to repeatedly churn out high-quality bug-free code, and to create an objective process methodology so that time and money is well spent. We may end up with multiple different methodologies for various technologies, domains, etc.
I understand what you mean, but I'd have to disagree with that. I've been working on a very large engineering project with a large-ish team (~50 members) for over two years. All of our communication is via an established methodology of engineering diagrams, design documents, position papers, etc all of which are in a structured format that follows common rules/regulations/conventions.
As a result, all the companies we work with for e.g. understand our P&ID diagrams, electrical schematics, mechanical design docs, system layout diagrams, etc. There is no reason why such a methodology can't be brought to code. I don't view code as anything special - having now worked on on both sides. I think there really is a lot of value in the engineering methodologies that can be adapted and applied to the software world.
In reality I think someone who only looks at team members who are "doing the most" is really just measuring who is talking about their work the most. You need some kind of objective measures like features shipped, assigned bugs resolved, etc.
Now the objective measure itself might have some sort of bias, but at least the rules are set and you're not getting screwed over based on someone's feelings. You can argue metrics, you can't argue your managers feelings.
Or how about even if yours generates 10 comments on the PR correcting things to match code quality guidelines and mine doesn't?
I don't think we often track things like that.
You have to define productive engineer in order to claim the metrics are biased.
I think it's impossible to create something like that and it'd be very very bureaucratic and still prone to gaming. I think having something 'in between' is the best approach. You still allow a manager to interpret these things together with you but the manager should give you a guideline for what to look out for. We can use these metrics to inform decisions about performance but it's completely counter productive to simply have a few metrics where you have to hit specific numbers.
PR throughput? No problem, I'll form a clique of a bunch of people that OK each others tiny PRs. This will result in so much overhead that we won't actually get much done, piss off other team members, create a hell of basically unusable commits, make it more likely that code quality suffers because nobody has any chance of having an overview of what you're doing overall and you will likely create regressions that developed over multiple commits and would've been caught otherwise because let's face it, each unit test you write is its own PR. You say obviously you won't get through with this because your manager is supposed to stop that? Well he can't if we just want a completely objective and metrics driven approach!
With the hybrid approach, you know from your manager that PR throughput is important not at the expense of quality and other things. You want small PRs for certain reasons but not at all costs. There is no exact formula because no two situations are exactly the same. Of course bias comes, of course bad managers make this bad. So does a completely "objective" metrics driven environment in which you play the metrics game. There is no perfect solution.
Replacing a subjective metric that's at least somewhat effective with a metric that's totally useless (because those inherent inaccuracies/biases are even worse than even a poor manager's judgement), that's throwing out the baby with the bathwater. The primary purpose of a performance metric is to measure performance, and being prejudice-resistant is something that's nice to have - the primary reason why you implement a metric is not because you need something that can be argued.
This shouldn't be surprise at performance review time, nor should it necessarily come from your manager -- it should be coming from your senior colleagues on each of your code reviews, giving you a chance to improve your bad code before it gets checked in. But when your senior colleagues think your PRs are worse on average than those of your peers, then yes absolutely you should get a worse rating.
Also, the metrics you choose will undoubtedly contain a measure of your bias anyways.
For instance, the metrics two people would choose to represent developer effectiveness will not be the same, and those differences will reveal what kinds of workers they prefer.
I think there is a conflict of interest there, though. Managers have a vested interest in saying that their team is highly productive. Managers of highly productive teams get raises and more head count, and eventually promotions. Anything else reflects poorly on the manager.
So the managers-of-managers do need to keep their eyes on this too, but I also agree with you that it's harder for people in that higher-level position to evaluate this. I guess, as you hint at, the manager-of-managers can look at team output overall, and if that's below expectations, that's a starting point for discussion with the line manager.
I was reading an interesting piece in the economist today about maintaining peoples performance on a long space flight. One part that stood out to me is people being productive makes them happy. I always assumed the causation would be the opposite way round. Perhaps the best a manager of managers can do is try and figure out if the members of each team are happy or not.
> Measuring developer outputs can be detrimental
And then:
> Design and coding: The number of design papers and specs, work items, pull requests, commits, and code reviews, as well as their volume or count.
All they do is add more and more metrics. But this has the exact same problem as with the infamous KLoCs measure: how do you interpret it? How do you know it is not gamed, to begin with? Actually, now you have two problems: collecting and analyzing this mass of metrics can have a significant cost.
Bug bounties work, why wouldn't "feature bounties" also work?
You say you want those features, preferably bug-free, for this deadline. And there's $5K for the team if the objectives are met.
Then, your metrics problem boils down to how to impartially measure customer satisfaction, or how well the objectives are met (in some contexts, bugs are unavoidable etc.).
Metrics can still be important to help the team identify their problems (or rather, confirm that their intuitions about the problem). It's an optimization problem: measure first, then do something about the actual bottlenecks.
That said, some programmers are such nerds that more money is not the highest motivation. One can use some creativity here.
In this case, you'll get your features, but they very likely won't be bug-free. They'll probably be quite slow and fragile. They might not scale. They might not be well thought-out. They might break backwards compatibility, or break other features that your customers are already using.
In other words, why wouldn't a developer borrow limitless technical debt in order to claim the bounty as fast as possible and move on to their next bounty?
On top of that you have the problem that now your developers will follow the set objectives to the letter even if it transpires that something else was required.
While I'm not a fan of subjectivity in ratings, the challenge is that it is very difficult, and I would argue, virtually impossible to do it objectively. So what happens instead is that when metrics are used to evaluate engineers, the smart ones figure out how to game them. Does that make them, or the team more productive? Nope. Can that have un-intended consequences that actually make the code less stable, and decrease productivity. Yup!
But if you're going to go with these measurements you might as well go big. Throw out anything related to Agile, require estimates that are accurate within 15 minutes and severely punish engineers for not getting estimates right. Might as well also add in heavy documentation requirements too. After all, this rigorous measurement etc. has all worked so well in the past <dripping sarcasm for this last paragraph>.
Rather than task the developers with all manner of bureaucratic Agile bullshit like tracking hours, arguing about story points, submitting to kindergarten-style daily stand-ups, velocity tracking, retros, etc; I would suggest a different tact. How about measuring developer productivity by observing if they're building what you need at the rate you need it built. If not, then you need to figure out if you can afford to replace them with someone that can.
This is a much more entertaining version of Goodhart's law [1]
Great review of the hazards involved in quantifying developer productivity - the correlation above has been true everywhere I’ve ever worked.
If the company you’re working for:
- is not investing in improving the developer experience
- is not listening to developer complaints about slow, tedious, or error prone processes
- is perpetually pushing tech debt onto a backlog that only grows
Then chances are you work for a company whose leadership does not understand and value software engineering. They likely see it as a cost center, and they likely incentivize managers by rewarding initial delivery of projects, at the expense of maintainability and developer sanity.
I know I’m preaching to the choir, I just had to put it out there for all the young engineers. Don’t waste too much of your life and happiness trying to patch those sinking ships.
> your life and happiness
Also remember that these things are why you're here in the first place, and not to be the best Level 4 SWE Management Trainee in the trans-western division this quarter.
It's only healthy to maintain perspective. It can hurt in the short run sometimes, but there's only misery if you don't keep it.
The research is also slowly laying the foundation of what are useful metrics to track and what excellence looks like for the industry. Unfortunately, those metrics are typically difficult to measure because the underlying data often spans multiple engineering systems: Lead Time, the poster child of DORA metrics, requires data from at least your source control and your CI/CD systems.
Btw, you might be interested in checking out Faros Community Edition: https://github.com/faros-ai/faros-community-edition – an open-source engineering operations platform we’ve been building for this very purpose. Our goal is to bring visibility into engineering operations, and make it very easy to query and leverage data both within and across your systems. It’s container-based and built on top of Airbyte, Hasura, Metabase, dbt, and n8n.
> Number of commits.
> Average commit size.
> Frequency of code reviews.
> Number of code reviews.
> Time to review.
> and so on...
No. This has been tried many times and companies think this is how you measure productivity, but it is not even sustainable. Developer productivity is not about moving the needle, it is about outcomes, and not outputs.
An outcome is finally merging an unsustainable PR that has sat for a month. It is not how many comments, reviews, meetings, or commits needed to get to the outcome.
The only people I know who want to implement these terrible measurements are the type of people who have ambitions as large as Mount Everest but die on the decline back down. The real goal is to be more like a f1 pit crew where you leave out the metrics and end up performing better than if you measured them.
What criteria does your team use to measure the outcomes and/or during a post-mortem?
> The real goal is to be more like a f1 pit crew where you leave out the metrics and end up performing better than if you measured them.
But all F1 pit crews have defined measurable metrics for success. I don't see the analogy here? Can you help me understand it?
The f1 pit crew analogy refers to how certain teams realized that measuring KPIs too religiously meant the vary difference of performance when it mattered.
Similar to other sports where coaching actually matters and these metrics are rather useless unless there’s a coaching role to be deliberate with them.
I see, so you guys don't measure anything specific that is actionable? Did you face any challenges dealing with non-performers or under-performers dragging the team down?
Ultimately the desired outcome is money arriving in the bank. Which is even less fathomable.
the only quality that matters is how effective the software is in its business function. how effective does it make stakeholders? how well does it capture engagement by users? the right question to ask changes in business context, but if you can't answer it, you might as well throw darts and flip coins. if you can measure the impact of their code before and after deployment you might have a chance, but it's probably hopeless.
as far as I can tell it boils down to a subjective and qualitative assessment of developer performance. you can also take the contrapositive: where would we be without this person? how long would we have taken to get there without them? what would we not have learned without this person?
I'm nervous about the implicit bias that comes with this kind of perspective, but I think it's the best we have for now.
Let's leave aside for a moment that this piece begins with an impressively uninformed and circular definition – "Developer productivity, in general, refers to how productive a developer is during a specific time or based on any criteria." – and focus instead on the question of why does this stuff keep popping into existence; what's behind it?
As a tech exec who's researched and given several talks on this to large audiences of non-technical execs like CEOs and CFOs, I believe the root causes are an understandable and intense desire for "visibility" and exec accountability coupled with a set of false beliefs held by non-technical managers including "anything can be measured if you try hard enough" and "nothing can be managed unless it's measured" and the classic quantitative fallacy of "things that can be measured are more important than things that can't be". Besides, it's only fair that if the VPs of sales and marketing have to stand up and talk about funnel metrics and sales rep productivity (with real metrics like net new bookings divided by fully loaded sales rep cost) that the VP of engineering - an often enormous fraction of a SaaS company's budget – should be similarly held to account for some number, any number, we just need a number, so we can look for "trends" (actually, noise). It also seems to be driven by a push from HR for fairness in promotions and terminations, which is also totally understandable, yet misguided.
I have a wisecrack response to non-technical executives when discussing this which is "how do you measure your own productivity?" that helps them understand the absurdity of what they're trying to do and how common it is that no true measure of productivity exists. People really struggle to understand that some metrics, no matter how great it would be to have them, simply do not exist, and so we have this – measurement theater.
[edit: fixed typo]
The questions of whether features are appreciated by users, or which bugs should be fixed or not, or if a product is feature complete or needs more, are questions of business, and not developer, productivity and efficiency.
And regarding documentation, I consider that an integral part of code/software that can be judged similarly w.r.t. quality and impact, having its own features and bugs.
Using merge time is a terrible metric, perhaps worse so than deadlines because it can be more effectively weaponized. What would the incentive be for the developers other than to rush the code review process? Merges are not where you want to be rushing anything, but rather the opposite. If project deadlines are necessary, allowing code review its due time affords developers the ability to informally schedule things without sacrificing craftsmanship for what in reality is a vanity metric. Some code needs the be carefully considered and given time while other code doesn't necessarily need much review or worry at all, but no one can tell that by looking at mean time. If a developer is asked why some tasks had a longer than average mean time, then now they have to waste even more time by explaining themselves. In the worst case, the incentive to rush the review process results in more time wasted on bugs that could have been caught before they even had a chance to be merged.
Am I misunderstanding your view of how mean time to merge would be used?
I'm currently experimenting with using "Mode time" as I think it is less susceptible to data skew from outliers. See example below:
https://oss.gitsense.com/insights/github?p=days-open&q=days-...
For popular open-source projects that I used in the link above, the Mode time to merge is less than a day which is quite good in my opinion. And as you can expect, if you look at larger pull requests (>=10 file changes) the overall percentage drops by half as the link below shows:
https://oss.gitsense.com/insights/github?p=days-open&q=pull-...
I think what the link above shows is, you can't just willy-nilly use merge time to measure productivity, since there are a multiple variables at play.
Full disclosure: The link that I referenced is my tool
I think that qualitative metrics like satisfaction and collaboration could be helpful, especially when combined with traditional metrics like mean time to merge a code change. Taking my previous example of overworking or cutting corners to achieve high numbers, a qualitative metric for something like satisfaction might indicate a problem where a work output metric wouldn't.
But I think that any combination of metrics will be an oversimplification that could lead to problems if they are the only thing that matters. I'm not sure where the balance lies. I like that metrics can offer an objective view of performance and make it easy to spot trends. But I am wary of them oversimplifying things and dehumanising the team.
I think this depends on how strongly leadership/management tries to use this metric to change behavior, but in a company like mine where we have a lot of business dependencies, contracts etc which rely on predictions of throughput, this is an important metric for us to predict timelines. It's not used against a team or platform and it is primarily used internally by management and not so much engineering teams which I think is the right way to use it.
Absolutely. It’s the old descriptive vs normative saw. We should be very interested in measuring how productive we are. But how do we really know how productive we should be?
I think trying to answer that is hard to impossible for a developer, because you can always automate or abstract further, but the cost to do so marginally increases and you probably won’t know the full cost until it’s already realized, at which point, requirement and prediction are obviated.
I know I’ve had my fair share of negative experiences with a scrum team that wanted ever faster velocity and used burndown charts as a stick with no carrot in sight.
I could see how a satisfaction metric would counterbalance a work output metric. The team will probably feel less satisfied if they know that corners are cut or if they are having to work unsustainably.
How about just doing your best as an organization and as a people manager to make your developers happy and fulfilled? That increases productivity and motivation to succeed more than anything, IMO. Give great pay raises regularly, give a ton of time off, get rid of people managers who are jerks, etc. If your company has goals, and your developers aren't producing code to meet the goals, your goals are probably too high, you have too few developers, or your developers aren't motivated to complete the goals because they are being treated like shit or don't agree with the goals.
Management always wants to think that they are right in every decision and the employees are the ones who are unproductive, but after decades of working for "the man" in about 10 different industries in different positions/careers, I have found the fault lies with management 80 to 90 percent of the time due to some leadership failure or combination of failures. The problem is poor leadership and lack of motivation, no doubt in my mind. I've also led large groups of people (in the military) and by far the best thing I could do for them was make their personal and work lives better by not getting in the way and by not acting like a dickhead. Adding metrics to things just caused more useless work for me. You can't force change in a system via metrics, the only place where measurement changes outcome is in quantum physics.
I hate to go on a "capitalism vs. communism" type rant, but the best places I have ever worked, with the best "productivity", have been flat orgs where the developers and other employees are included in the decision making and the management and execs are open and caring and don't try to put profits and the business above the personnel. When everyone shared the success or failure of the company on equal terms, we could all get things done that were unthinkable.
Getting crap reviews/evaluations because a project failed due to management screwups is the universe's way of saying "you should have left long before, but leaving now is your best available alternative."
Much of this discussion is how to protect people from the consequences of staying in a bad situation.
The correct response is to not to try to fix things, but to leave. Staying only perpetuates the problems.
Starve the beast.
Well, schedule and cost at least have straightforward measurements.
The issue then is features. Or is that it?
The "pick two" model really is just the business view. Invariably you will also have:
- adherence to process (ideally process would be an overall enhancement to productivity, but it usually becomes a net-negative)
- maintenance costs (patching, libraries, language versions, database versions)
- infrastructure churn and upkeep
- random org shit: meetings, more meetings, training, certifications, HR, ticket walls, etc
- documentation. Is that important?
- ... are the requirements known? settled? at least ballparked?
As I see here, invariably measuring developer productivity is of course blaming the victim: WHY AREN'T YOU MORE PRODUCTIVE, and of course shrugging away the nigh-unlimited ways an org can hamstring or frustrate a developer.
The reality is always in the middle.
Example: If you solve a problem quickly that leads to support ticket being open that's not good.
Possibly the world expert at this point (Dr Nicole Forsgren) in this exact topic comes up with a framework based on the best of what she knows from years of studying this and refining her approach.
Random HN commenter: ahh just measure time to commit.
Random HN commenter: biases are cool, so just use personal judgement.
> The DORA (DevOps Research and Assessment) framework introduced some metrics to track team flow, such as deployment frequency, which measures how frequently an organization successfully releases to production, and lead time for changes, which measures how long it takes a commit to reach production. If you're interested in the DORA framework, we published a dedicated article on How to implement the Four Key Accelerate DevOps Metrics.
(you're still not wrong tho haha)
Fix a bug, Add feature set Do something new.
My epithet for one programmer was "He writes a lot of code"
Extra code makes it harder for the next guy to figure out what's going on, and has more space for bugs.
But that came from a "productive" developer and that code can tie down a dozen maintainers in dozens of customer sites.
The productivity is job creation for a bunch of folks whose main ambition is finding a job where they don't have to work with crap code.
I've done a number of projects where I got rid of multiples of code compared what I put in.
The best example was where I replaced a subroutine with a single character constant.
This all falls apart without a good tight spec. If the spec is loosy-goosy, then forgetaboutit.
Maybe I've had 30 years of bad luck, but I've never seen a "good tight spec" since I started programming professionally in 1992. Most of the time there's no "spec" at all.
Even if you do manage to get the estimate-demanders to back off until the spec is good and tight, you're just moving the problem upstream - they'll just want an estimate on how long it will take to get the spec right.
>they'll just want an estimate on how long it will take to get the spec right.
The people who want the estimates are the same people responsible for the spec, so you're actually pushing the problem onto where it belongs.