Worst Software Metrics
gitclear.com
gitclear.com
I wish I could show management how negatively and immediately that move affected the culture & mood of engineering at that company. It was an overnight change, and going forward it basically killed all trust in management. I stayed around for a little bit longer, but I should have quit the second they started those measurements.
It absolutely murdered the confidence of our junior devs. Of the senior engineers who were put on it, none of them really had any idea what they were doing wrong or how to improve, as no feedback was given about any of that, just that they weren't enough. Some of our seniors had 15+ years of experience in software and were industry experts, but it didn't matter: They hadn't been hitting the number of commits and pull requests and tickets per week that management insisted on.
When management was confronted about it, they admitted that code metrics aren't a good measurement of productivity, but then insisted that they "needed something".
It turns out, they only used code metrics as a bludgeon to get rid of the employees they wanted gone, and since then that's all I've been able to see code metrics as. They're a battering weapon used to give HR a performance reason to justify firing you.
I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.
And by the way this isn't sour grapes, metrics have almost always made me look good. One job I left, I had so many more commits than the next person that it took them two years after I left to catch up to me. And yet I'm absolutely certain I was not the most useful person on that team. The most useful person was the guy that was writing the extremely sophisticated modelling kernel, and if you only looked at the metrics he probably looked borderline useless, but in actual terms he was by far the most valuable and important person on our team.
What scares programmers is, everyone on our team knew that, and any sort of manager with programming experience probably knows that, but you know that those stats are going to eventually float up to someone who doesn't know that the numbers are bogus. (After all, they wouldn't be tracking these numbers if they knew how bad they are). So decisions do get made from these numbers, even when people promise they wont.
Now, this comes with interesting points: - the metrics in this case are in the hands of the professional in the field - the metrics are there to serve the person itself, without affecting others
Metrics on commits, line of code, tickets closed are useless from my point of view. If you ask in a developer team who is weak at the moment (assuming there is no room for improvement is stupid) gives better results. A decent programmer is able to spot fake ones and is able to notice performance difference over time (not perfectly, but definitely better than metrics).
Not to mention, I would be surprised if none of the team would start complaining if someone is doing no work and still getting salary.
You can spend an entire day teaching your team something important about the project you are working on, have no commits about it. Or you can use half the day walking around thinking how to solve a bug. Not a single line is written, but that's how some problems are solved.
I started counting in my working hours even when sometimes I wake up in the middle of the night and my mind solves programming problems for 1 hour (let's say). I have a decent employee, being at my desk typing is not what I'm paid for, I'm paid for solving problems with software engineering to the best of my knowledge.
I currently work at a company where all the devs are pretty good and the code is decent. We don’t really need much in the way of metrics. If anything, we have more metrics than we should. I trust my current team to do the right thing more than I trust a linter.
Luckily all this useless theater only took up an entire day every two weeks. If you assume everyone on the team was making about $50 an hour, and it was about 6-8 people in attendance to these things, then it came out that this very dumb "estimation" process took 6 people * $50/hr * 6 hours... so about 1800 dollars. (Not including the project managers salary, which given that he drove around a very fancy luxury car, was probably a lot more than the rest of us).
And people wonder why I don't take scrum seriously.
dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6.
pm: no 6 is not allowed, it must be either a 5 or an 8.
dev: ok, lets call it an 8.
Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.
Personally I prefer being on a small team with excellent devs that don't require this kind of stuff. I built products that made many millions of dollars a year with just me, a PM, and one other dev with none of this stuff.
So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybody measured if these measurements actually are helpfull?
The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm unaware of any study that addresses the question.
One possible workaround would be to take experts in the field, and try to translate their qualitative interpretation into input that drives a metric. e.g., if Andrew Clark from React could list three directories that harbor disproportionate tech debt in their project, we could calibrate our Open Repo Directory Browser to identify the factors that were unique to that directory. This could then help managers pick out when and where its appropriate to make payments to reduce their tech debt.
Unfortunately, the prominent open source developers aren't answering my emails at this point. And even if they did, it's still a long road to translate qualitative expert opinion into provable long-term benefits. But I think the environment for making that leap is as ripe as ever. We just need to keep working toward figuring out the most desirable+measurable outputs.
The metrics are all an effort to abstract developer output into something a manager with no/minimal domain experience can understand.
It feels very similar to the measurement/quantification problems distorting academic output.
Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.
The problem is, the business people (generally) have no interest in this. They're interested in certainties. They aren't being irrational either -- they need to go out and sell, raise money, etc., and to do that they need to be able to say "we're shipping on december 5th" and "absolutely we can do that for you". Having the developers say "well, uh, maybe we can do that in three months but we don't really know" makes their lives 10x harder even if it's true.
So this is where project managers come into the picture. They don't promise the most efficient use of human resources. What they offer is a predictable use of human resources. In reality this drags everyone down into a very gloomy mediocrity and leads to ineffectual organizations. But to the business person, it seems great because they gain the illusion of control.
In that light, our current predicament makes perfect sense: project managers are the shamanic priests doing a rain dance, and developers are the weather. The weather stays unpredictable, but if it happens to rain the shaman can take the credit, and if the drought remains, it's because the ritual wasn't done correctly, you see. The illusion of control is the product they're selling. What the managers are buying is someone that says "yes, we can have it rain on the 19th" instead of talking to someone that says "we can predict the weather about a day out, maybe"
(God I hope no project manager I work with ends up reading this, heh).
Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another).
Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.
Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry.
I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus.
Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs."
Bad Incentives --> Bad Outcomes
I worked at a healthcare company where the chief architect was considered a genius by upper management because he would dive in and "save" production; in reality he was the cause of all the bugs that broke it but he did not allow anyone to fix the problems.
Even worse, the typical braindead “solution” to this problem is to insist that everything be broken up into chunks of a few hours or less each before work can begin. The end result is “if it can’t be done in a couple of hours, it isn’t worth doing”.
- that complexity metrics are highly correlated with LOC.
- people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind.
Both facts are discussed at length in "Making Software" (Oram, Wilson)
>Thus, the "most valuable developer" as measured by LoC tends to be whoever is adding the most CSS, whitespace, and third-party libraries.
It's still as terrible as ever to positively correlate developer productivity/quality/skill with LOC.
Though, agreed you are almost certainly better pulling back up and asking more product faced questions. The most prolific author, as an example, produced more sentences read by someone. Not just more sentences.
* In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it.
* In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful
* In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving
* In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward
I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.
Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n.
(I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/)
It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.
My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.
* In baseball, batting average and OPS require you to compete against other teams.
* In finance, ROI is tested in the marketplace.
* In sales, conversions require you to convince a skeptical customer.
The software metrics are all things we have fairly direct control over.
Maybe that suggests an amendment to Godhart's law:
> When a measure that is controlled by the measuree becomes a target, it ceases to be a good measure.
I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired outcome and attempt to implement a system that teams can use to apply the metric to their work. It ends up being effectively impossibly complex to create the formula, and even if you could create it, it would take longer it than it would to write the software that it's supposed to measure.
What we're left with is 'the best that we can do', and that's what we have been doing. We absolutely COULD make more accurate(yet still very imperfect) metrics that more closely model the actual goals of a project by simply including more factors(bugs, readability of code, difficulty of feature, size of feature/LOC, number of modules/classes that the code spans or communicates with in some way, how easy-to-read and easy-to-modify the existing code that needs to be altered is, etc times 1,000,000). But adding more factors just makes the metric exponentially more complex and difficult to implement. I'm sure people have tried using way more fine-grained metrics than what we have now - but nobody knows about them because nobody uses them anymore because they weren't worth the time and effort.
So we're kind of left with what we have, which in my opinion as a software engineer is definitely not good enough. Which is pretty bad, because by 'good enough' I just mean better than doing nothing at all.
The only problem here that can be solved, and therefore that is worth solving, is why these metrics are still commonly enforced despite being counterproductive. Obviously managers managers-of-managers feel a need to have some nice, concrete, objective numbers that they can look at to see how well developers and teams are doing. The people pushing these metrics are often the ones who have no other way of understanding what exactly developers are doing, either because they've never written software before or because they are too far removed from the level of the guys actually writing code to pay attention to what they're actually doing beyond just looking at a single number.
It makes sense for managers who don't know what engineers are doing to seek out a different way to measure their performance and progress. The solution isn't do nothing, and it's not story points or any of that BS. But what are the alternatives?
The only one I can think of is, don't put a manager in a position to manage individual developers if he/she is unable to understand what those individuals are even doing. Having an experienced senior engineer with the right skillset for management directly in charge of every team is amazingly valuable in my opinion. It's also vital that those higher up on the ladder are able to trust these manager-engineers, who are basically acting as an abstraction layer between the regular engineers and higher management.
A lot of teams already have this in the form of a senior engineer or software architect, my team has one and it works out amazingly. We all feel that doing a good job will be recognized and rewarded, problems are easily identified and addressed, we get things done very quickly, everyone gets along, and there's not a lot of pressure.
This is just my own single anecdotal experience and it has worked out very well but I'm aware it may not always be that way, so I'm interested in hearing others thoughts/experiences/solutions
Managers are engineers of towering technical competence.
That is shocking and horrifying if that's the norm for software development.
Lines of Java + TypeScript, my two most used languages, purposefully excluding configs, html/css, etc. Assuming 8 hours of work per day; this is less than the time I spend at work, but probably more than I spend coding, due to meetings/email/overhead. 246 workdays, which is 30 days off, some of which are public holidays and the rest are vacation. I'm a relatively junior dev.
One thing that I think can give you perspective is if you experience a job where you account for your time like lawyers do, in 6 or 15 minute increments. It's surprisingly hard to get your billable time over 50%, assuming you're honest about it.
My position is that if you're evaluating people by comparing metrics, you're probably a terrible manager.
If we could channel energy from creating performance metrics into a form of reporting that continually narrows down the problems that need solving, productivity would be served better.
Otherwise we could probably use keystrokes per second just as well.
The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments.
The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores.
After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code.
FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.
It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer.
If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get.
We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement).
Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science.
We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't.
There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.
With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.
"Context, action, outcome" is a sufficiently standardized formalism for evaluating performance (assuming outcomes are measured in terms of business value). Accurately reporting context, action, outcome requires understanding both what devs do, why they do it, and how it ultimately produces business value. There's no shortcut. If your management can't meaningfully do that because they are non technical or managing too many people, or whatever, then they are actively harming your team and should be removed with prejudice.
There is no reasonable way to conclude that devs can be effectively measured with garbage stats. Even suggesting it seriously indicates incompetence or laziness.
I completely don't get this weird iframe for the entire article thing.
If someone has a very low number of commits relative to others on their team, I’ll investigate deeper - review a few Pull Requests, check if they’re working on something outside source control etc. sometimes I find problems, other times not.
I’m curious what people think of this approach?
I do generally try to enforce the goal that on any working day there should be at least one commit from the developer. If I notice a developer is consistently missing this metric it usually indicates trouble in the project they are working on or that the developer is just not delivering enough.
Also, beware of pretty graph summaries that are presented without a healthy list of caveats and follow-ups and further explanations. Beware if no one with management power is asking the follow-up questions that matter.