Hacker News Reader

Comment by geekjock | Hacker News Reader

"What should we measure to improve developer productivity?" is a decades-old problem for leaders with no clear solution.

There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach.

Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity.

A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com

View on HN

> "What should we measure to improve developer productivity?"

How about nothing? Honest question. I am sick and tired of being treated like cattle. I've never seen a single convincing argument that any metric of my outputs has mapped to some productivity metric in a meaningful way, but I have absolutely seen them weaponized against me and others to punish and fire engineers. All of these metrics have only ever boiled down to a negative reinforcement mechanism, in my experience.

I'm a developer and am right there with you. But if you're a decades-old corporation with 10,000 engineers, you need some set of signals to help guide improvements to tools and processes, right? This benefits developers, and there should be a set of signals to enable this.

You need objective, measurable signals if you are concerned that developers might be confused or lying. Otherwise you can just ask us. Most people are happy to tell you what the pain points are in their workflow and how they compare to last year's. Maybe you need metrics to quantify specific complaints like "slow" or "crashes a lot." But I feel like most organizations doing this kind of measurement are looking for some kind of Freakonomics "developers think they want X, but what makes them better is actually Y" when they haven't even bothered to ask about, let alone implement, X.

> You need objective, measurable signals if you are concerned that developers might be confused or lying. Otherwise you can just ask us

In an organization with hundreds or thousands of developers, there will be people either lying about how productive they are or genuinely think they are performing above average when they are not. It's like how 80% of people think they're above average drivers.

On the other hand, you may have excellent developers that are overly modest or not loud enough to sell themselves. They are truly exceptional and above the curve and should be rewarded for that. Some of the 20% of drivers that don't think they are above average may actually be above average.

Objective, measurable signals help to find the outliers at the ends of the curve that may otherwise be missed.

It doesn't matter what percentile developer you are to guide improvements to tools and processes. It matters what slows you down.

In healthy orgs you are collecting such metrics to help advocate for investing in tooling, automation, infrastructure, process improvements etc. It can give the business case for building/expanding dev tooling and cloud teams. I’ve found it helpful in past jobs… fully weighted dev salaries are so high, at a moderate sized org a little bit of rough efficiency math can make for obvious investment cases.

> In healthy orgs...

This is, so far as I can tell, ~5% of them.

IMO some ~objective feedback of reality is good though. Probably not easy to set up without having it quickly turn into a negative reinforcement loop indeed. But let's not forget that, with the right supporting culture, it can be helpful (pleasant?) to the dev to have some metrics. I admit "how to do that?" remains unclear.

So how do you deal with bad developers? How do you even measure if they exist in your organization.

And as much as many of us on HN are developers and like to toot our own horn, some of us suck and don't get better with time. I mean in small organizations this is typically easy to figure out, but in larger organizations it's a problem that can persist for long periods of time.

Who cares? No company I've ever heard of has been killed by bad developers. The same can't be said for managers, Vice Presidents or the C-Suite.

This is a tautology if I've ever heard one.

If a bad manager keeps only bad developers around it will kill a (software) company just as fast as anything else. It turns out that paying customers like their software to work and be free of data corruption.

I disagree. Under good management, bad developers might progress more slowly than good ones, but they won't outright destroy the product.

Speed == probability of survival for startups. You only have so much runway; better developer productivity means more swings of the bat at finding PMF.

That isn’t what is in contention, though. The mapping of some objective metric to that speed and productivity is what I’m arguing doesn’t make sense.

What we are arguing is the Chinese room problem coupled with Goodhearts law.

The people that need to interpret the metrics might be a professional that understand their meaning, or they might be an AI following arbitrary instructions. The first group will use the metrics to create a better team, and the second will make people angry.

When you lack good developers stuff like this happens:

https://www.cbc.ca/news/business/sony-cyberpunk-2077-playsta...

Edit with more arguments:

You could of course blame managers for that, but delaying a game since your bad developers aren't capable of delivering isn't really an option either. How many years will it take those developers to get things right? What they would need is to delay the game and replace the developers with good developers, or scale back the complexity of the game to a level those developers can handle.

The company in question choose the latter, they said they will use unreal engine for future games instead of an inhouse engine. So they admitted their developers aren't good enough.

> "So they admitted their developers aren't good enough."

Build vs. buy is pretty much never a question of developer quality, but a question of ROI. Even with the best devs in the world, building something on par with Unreal (or other commercial engines) is a huge outlay of capital. With relatively little return over a single game, you need multiple hits to pay off the engine.

There aren't a lot of companies that can afford that.

Here's a decent write-up of some of the major issues that actually plagued that project, as opposed to "bad devs": https://blog.devgenius.io/cyberpunk-2077-through-the-lens-of...

> Here's a decent write-up of some of the major issues that actually plagued that project, as opposed to "bad devs": https://blog.devgenius.io/cyberpunk-2077-through-the-lens-of...

All of those issues were there for their last project as well, but somehow they only failed this time even though they had a much larger budget and larger team with more developers. The most reasonable explanation is "bad devs", even though that will never be an official reason. When you swell the team size quality tend to take a hit, and when the old good devs becomes managers and the new devs try to replicate the work you often get problems.

That article describes how you need to work if you have bad developers. CDP-R assumed that they still had as good developers as they had previously, which wasn't true and that bit them.

"scale" is a problem. It is not the same problem as "bad devs".

All the indicators point to "scale". Having worked in games for almost two decades and having seen anything from solo dev to teams of hundreds, and the growth in between, my experience leads me to say "scale" as well.

Do you have any indicators for the "bad devs" problem? So far, you're just stating it as if it were categorical truth.

The big problem with this line of reasoning is that good developers hate working with bad developers. So if you don't do something about bad developers (which might just mean train them, not necessarily fire them), then the good developers will leave.

In that case, the bad developers are easy to spot. The good developers point right at them. There's no mystery nor are measurements needed.

That is circle reasoning, if you know which developers are good so you can ask them then you already know which developers are bad.

And asking good developers to identify good developers doesn't work either. Good developers hates working with bad developers, true, but like all other humans they hate working with all sorts of people for different reason so they will point out a lot of good developers as well.

Instead what works is if you have lots of data from diverse group of good developers. That way the personal noise gets cancelled out and you are mostly left with the "I don't like bad developers" signal. This isn't trivial data to get though, especially since people usually choose to work with people like themselves so a team likely isn't diverse enough to get you this kind of data, and people from other teams aren't familiar enough to give good data either.

You hire 5 developers.

3 of them point at 2 of them and say they are bad.

You fire 2 developers.

Nothing gets done. Turns out the 2 were the good developers.

Congratulations, you succeeded at making another bad metric.

That's just bad statistical analysis.

Paying a non-trivial portion of your engineers high salaries will kill any startup without unlimited money, regardless of what other guardrails you have in place.

> So how do you deal with bad developers?

The same way we deal with good developers? With managers, of course. That's the whole point of their role. Every developer productivity metric I've seen speaks to some disparity between engineering and the C-suite. I don't understand where these products originate that seek to do the manager's job for them but worse. I've been in management roles before, the numbers don't and can't always tell you the full story, no matter the size of the team or the quality of the data.

As always, good people are hard to find.

What if the manager is bad? Metrics can help other people identify that a bad manager may not be identifying very good developers.

If the manager is bad, we should address that by discussing metrics to measure manager success and identify three bad managers, work with them on improving and fire them off it doesn't help.

> So how do you deal with bad developers? How do you even measure if they exist in your organization.

Ask the developers that work with them? Who likes working with someone that isn't pulling their own weight, or worse, that is just making your own work harder?

But if you don't measure them—or if, god forbid, you give them their own offices with doors—they might start to think they're in the professional/upper-middle class, not the middle class!

This doesn’t work, because then your company would have to measure your value by the value you deliver. That would probably mean having to compensate people like you way better, because software engineers can create immense value. Of course this can’t happen, as we apparently live in a universe where only management have bonus based compensation.

Who’s we? FAANG engineers have bonus and stock compensation.

“the things that slow them down or frustrate them”

Whenever management makes a new attempt at getting more precise estimates I always tell them the only way to get things done is by doing them. So the number one goal should be to remove obstacles, simplify processes and make sure tools work to our benefit. For some reason nothing much ever changes and the question for better estimates returns. It’s pretty weird.

Anything that is routinely easy to estimate can probably be automated away. And so will your competition.

Reply on news.ycombinator.com