How about nothing? Honest question. I am sick and tired of being treated like cattle. I've never seen a single convincing argument that any metric of my outputs has mapped to some productivity metric in a meaningful way, but I have absolutely seen them weaponized against me and others to punish and fire engineers. All of these metrics have only ever boiled down to a negative reinforcement mechanism, in my experience.
In an organization with hundreds or thousands of developers, there will be people either lying about how productive they are or genuinely think they are performing above average when they are not. It's like how 80% of people think they're above average drivers.
On the other hand, you may have excellent developers that are overly modest or not loud enough to sell themselves. They are truly exceptional and above the curve and should be rewarded for that. Some of the 20% of drivers that don't think they are above average may actually be above average.
Objective, measurable signals help to find the outliers at the ends of the curve that may otherwise be missed.
This is, so far as I can tell, ~5% of them.
And as much as many of us on HN are developers and like to toot our own horn, some of us suck and don't get better with time. I mean in small organizations this is typically easy to figure out, but in larger organizations it's a problem that can persist for long periods of time.
The same way we deal with good developers? With managers, of course. That's the whole point of their role. Every developer productivity metric I've seen speaks to some disparity between engineering and the C-suite. I don't understand where these products originate that seek to do the manager's job for them but worse. I've been in management roles before, the numbers don't and can't always tell you the full story, no matter the size of the team or the quality of the data.
As always, good people are hard to find.
Ask the developers that work with them? Who likes working with someone that isn't pulling their own weight, or worse, that is just making your own work harder?
And asking good developers to identify good developers doesn't work either. Good developers hates working with bad developers, true, but like all other humans they hate working with all sorts of people for different reason so they will point out a lot of good developers as well.
Instead what works is if you have lots of data from diverse group of good developers. That way the personal noise gets cancelled out and you are mostly left with the "I don't like bad developers" signal. This isn't trivial data to get though, especially since people usually choose to work with people like themselves so a team likely isn't diverse enough to get you this kind of data, and people from other teams aren't familiar enough to give good data either.
3 of them point at 2 of them and say they are bad.
You fire 2 developers.
Nothing gets done. Turns out the 2 were the good developers.
Congratulations, you succeeded at making another bad metric.
If a bad manager keeps only bad developers around it will kill a (software) company just as fast as anything else. It turns out that paying customers like their software to work and be free of data corruption.
The people that need to interpret the metrics might be a professional that understand their meaning, or they might be an AI following arbitrary instructions. The first group will use the metrics to create a better team, and the second will make people angry.
https://www.cbc.ca/news/business/sony-cyberpunk-2077-playsta...
Edit with more arguments:
You could of course blame managers for that, but delaying a game since your bad developers aren't capable of delivering isn't really an option either. How many years will it take those developers to get things right? What they would need is to delay the game and replace the developers with good developers, or scale back the complexity of the game to a level those developers can handle.
The company in question choose the latter, they said they will use unreal engine for future games instead of an inhouse engine. So they admitted their developers aren't good enough.
Build vs. buy is pretty much never a question of developer quality, but a question of ROI. Even with the best devs in the world, building something on par with Unreal (or other commercial engines) is a huge outlay of capital. With relatively little return over a single game, you need multiple hits to pay off the engine.
There aren't a lot of companies that can afford that.
Here's a decent write-up of some of the major issues that actually plagued that project, as opposed to "bad devs": https://blog.devgenius.io/cyberpunk-2077-through-the-lens-of...
All of those issues were there for their last project as well, but somehow they only failed this time even though they had a much larger budget and larger team with more developers. The most reasonable explanation is "bad devs", even though that will never be an official reason. When you swell the team size quality tend to take a hit, and when the old good devs becomes managers and the new devs try to replicate the work you often get problems.
That article describes how you need to work if you have bad developers. CDP-R assumed that they still had as good developers as they had previously, which wasn't true and that bit them.
All the indicators point to "scale". Having worked in games for almost two decades and having seen anything from solo dev to teams of hundreds, and the growth in between, my experience leads me to say "scale" as well.
Do you have any indicators for the "bad devs" problem? So far, you're just stating it as if it were categorical truth.