Moneyball for Software Teams: Quantifying Dev Performance
software.rajivprab.com
software.rajivprab.com
Developers can see through this sort of bullshit.
"What can I do to unblock you and help you achieve your full potential?" is just a passive-aggressive way of saying you are unhappy with their performance.
Instead of this patronizing crap, how about the 1:1 goes like this: "How have you been? How is everything going for you and for the team?" and then just fucking listen?
The author here is charting velocity over time (down to the indivual developer, which is even worse) as some sort of work output proxy measure. This is a really big mistake. The entire premise for estimating stories in the first place is because we agree time is limited, so scope must be the variable. So you estimate stories, then you fit them into a fixed time period, then you execute and review the period. Did you get done what you planned? Probably not, why? I'll bet something turned out to be more work than you originally thought, or maybe you were blocked by someone else, maybe not even in the same team or even department. How did this period compare to previous periods? Is the planned/actual getting better or worse? Why?
As a manager I hope for steady velocity; it means the team is getting good at predicting the effort for work, probably understands what they're doing and allows you to make some plans for the future. People don't change the way the author's week-to-week velocity chart does, so all that shows is that they're still shitty at estimating stories.
One take-away that should be called out explicitly: velocity is a team-specific metric. Just as you can't divide it onto individuals, you can't compare it across teams either. Whenever you do either of these you're guilty of trying to inappropriately use it as a false proxy for performance. Please don't.
I'll say it again: software developers assist organizational learning. They don't produce things.
Which is what all those factory jobs used to be seen as, too. Much as I dislike Taylor, that's the attitude he was fighting against and he showed it was untrue. Same independently with Ford.
> just rehashing ideas from the past
Back atcha!
So here we are with two entrenched groups of people who "just know" that they're correct but are unable to prove it to one another. What's next?
Whenever I'm presented with a stark dichotomy, I assume there are other branches that people are ignoring.
Obviously (!) it's a bit of both, and mostly it's just like any other job nowadays. Full of its own mysticism, a ton of idiosyncrasies (but everyone has their own interpretation of which parts are exactly constitute the bad ones or the good ones), a huge ecosystem full of variety across many spatial and social dimensions, from code for microcontrollers for 1 dollar toys and fire and forget missile controller code to hyperscale web search engines and ChatGPT, GFW of China, and so on.
Just as every skyscraper, bridge, factory looks similar they are still unique. The sites, the soil, the permitting environment, the people involved almost always result in a similar yet unique mix.
And some similarities can be exploited, standardized, homogenized and treated as uniform or even virtually identical parts. From nuts and bolts to battery manager chips and k8s APIs and whatnot. But some similarities hide the classic devil in their details.
Of course the whole "industry" is in so much flux that it's prone to super embarrassing category errors (picking/selling the wrong tool, wrong platform, wrong experts) that end up costing a lot of money, lead to classic project failure modes (angry client, resentment, death marches, waste of expertise, crisis management meetings, sunk cost fallacies and aborted projects).
This regularly happens in many other jobs too. Again, the more megaprojectish something the more it'll go wrong like almost all megaprojectish things. ( https://en.wikipedia.org/wiki/Unfinished_building ... https://medium.com/@interestingshit/the-8-tallest-abandoned-... )
There are likely very few iron laws of project management, but the overhead due to impedance mismatch between teams, members, and requirements is probably one of them. It simply follows from this that any project that has either new teams or requirements is gambling on this, and as long as folks (up and down the hierarchy) are not up to speed on the domain they'll continue to roll the dice. And depending on the situation and luck we sometimes see mighty implosions (healthcare.gov, cyberpunk 2077, google stadia, and https://en.wikipedia.org/wiki/List_of_failed_and_overbudget_... ) and sometimes it seems like smooth sailing.
Software is codification of knowledge about domain and business process. If you wanna endorse and foster some form or another of cajoling it into making overtly complicated products for a boost on quarter revenues, we're just fundamentally of different world views.
I endorse healthy workplaces, hobbyist endeavors, coding challenges, navigating the murky waters of customer-supplier relationship with integrity, and absolutely despise how awful they are in many-many-many cases, and yes, it's mostly because of the absolutely fucked up incentives, like squeezing short-term revenue.
More domains than you would expect benefit from highly innovative work, and get less benefit from simply writing code as fast as possible.
Infoquake/Jump 225 series had an amazing fun-in-a-dystopian-way visions of this Taylor-ian model of programmers percussively hammering away on code.
Velocity is useful as a team measure, for the team to take on the appropriate amount of work for sprint. It is a measure to assist them predictably delivering work requested by their stakeholders.
Using as a statistic to be improved by an individual? I don't even.
TLDR, Kanban.
Maybe life is supposed to be satire.
The fact is many developers are treated like factory workers except they expect us to fill in the gaps.
Because most people are just going to say, “Fine” and try to not rock the boat. If their manager is passive they’ll be passive back. At some point the manager has to tell the report that things aren’t going well.
This also assumes that the manager isn't the blocker. Replying "Do your job better" probably wouldn't go down very well.
Sometimes you do have to help managers help you, and that's fine. But better for the IC if the IC brings it up before it's something showing up on a velocity chart...
> What can I do to unblock you and help you achieve your full potential?
We're humans not machines. Is everyone really that consistent? So if you did more the last 2 weeks and did less these 2 weeks, are you "blocked"?
It's great to have stats, but most of the time it's measured and used wrong by management.
How does velocity help a more senior engineer? Do you add in the number of times you've helped a junior or another team? Is that a blocker now? Are you going to unblock me by killing off the others!?
Why do I have to completely finish a task end to end before working on another 1? I'm not blocked, but just don't feel like doing this part for now. Is that fine? Clearly not!
It ignores all the human aspects of things.
What can I do for you?
The company favoring big impact project does not have any relation with a day to day agile methodology to control a dev.
"I would also pull up that individual’s personal velocity chart and use that to guide our performance"
Using a 1-5 scale for effort estimation should be a huge RED flag. It shows the author does not understand complexity and required effort do not scale in a linear way.
>> Break down large tasks into smaller subtasks, each of which can be easily scored using the above benchmarks.
...except it's the breaking down which is the hard part; this is not easy. Then adding all the sub components back up? This assumes there's no cost or effort; it's like saying lego kits assemble themselves.
>> every two weeks during my 1:1 discussions with each engineer, I would also pull up that individual’s personal velocity chart and use that to guide our performance discussion.
This is the definition of turning a 1:1 into a status report. The author would be better served by figuring out how to actually run a 1:1 rather than build a system where "Our methods certainly aren’t perfect and their flaws are obvious"
>> Velocity can then be quantified at both the individual and team level, by calculating at the end of each week the number of points that were completed in that week
>> Because the above methods are so widespread...
No, this is not a widespread method, and totally flawed premise. I've got news for the author: velocity doesn't measure progress, and it's not some sort of proxy for how much stuff Developer A finished vs. Developer B. At its best it shows if the team is getting better at scoping stories over time. At its worst you get garbage like this.
Managers will quickly game this by making a tons of useless small changes of button positions as this will give the manager a lot more velocity points than any real progress on the project.
> …to challenge the team to scale new summits in the coming week
> … If their chart is on an upward trend, I would congratulate them on a job well done.
Let’s use some basic logic here. If an engineer was working hard and smart last week and the week before, you’d want their chart to be flat, not going up. Or to put it another way, if you have team members whose chart keeps going up, what does that mean? Have they been sandbagging all along, and are gradually ramping up effort? Are they learning to game your system? These are not positive things.
In one part we have:
> if their chart is on a downward trend or significantly lagging behind their teammates
And in another we have:
> you should explicitly avoid comparing them against one another.
So don’t say you’re comparing them against each other, but definitely compare them against each other. Guess what? Sooner or later people realize how you’re BS’ing them, and how they just think you’re duplicitous.
The real way to manage teams is to (1) ask them for what you want; (2) listen to what they say in response and take it seriously (BTW, if something they say doesn’t make sense to you, it’s probably you who isn’t getting it — keep working at figuring it out).
You don’t need, and shouldn’t use, any numbers at the team level inside the team. As companies get larger numbers are necessary, but above the team level. (As a manager in a multi-team environment, you are going to need to play the numbers game — that’s going to be part of your job.)
Hm. Well, using a calibrated estimation (story points) is very useful on the long run. Mixing up its use as a descriptive predictive tool with a normative enforcement framework will lead to having a bad time :/
I'd manage the project and people differently, such that I don't think that interaction would ever take place.
(If I'm confronting an employee with a BS-process metric, that probably means that evil lizard people have seized control of the company, they'd directed me to make the employee quit, and when I refused, they kidnapped my family. The employee, who knows I would never do this willingly, would immediately deduce that evil lizard people had taken my family, and would quickly assemble the rest of our team, for hostage rescue.)
If I was your employee I'd be suiting up in tactical gear and buying business-suit-piercing rounds.
If you’re a low dollar contractor on a work visa making $25/hr to sling J2EE or COBOL for the Arkansas welfare department, that’s what it’s like.
If you don’t play the game, you stand to lose a lot.
Let's say I'm in the crappy scenario from TFA, and I have to be able to report the employee velocity metrics to upper management. I can still be honest with the employee about it.
Maybe one conversation would go like this: "You and the rest of the team are all doing great work together. There's a process thing I've been asked to do, which I wanted to get your input on. It's individual velocity metrics, and pressure to have the charts trend like so. Thoughts? ... Understood, and we're going to have to see how this works, and give feedback. For now, ideas on how we can get metrics trending in the desired way, honestly, and without significantly compromising our work? ... Are you OK doing that extra process? ... Will it save your time to offload the Jira part to me, and you just tell me the numbers? ..."
I say “pretend” because no matter your intentions, you’re a straw boss, and eventually you’ll toe the line or separate. Remember the supervisor is stuck there too.
In that example, seems like the employee was getting the most accurate available information about what the situation was.
If I was part of a solid team, I wouldn't know when to decide "Well, we've been invaded by evil lizard people, so time to start being jerks." I'd still be all about supporting the team and its work.
Besides doing our work being the default, I'd keep in mind that some team members might not be able to flee the evil lizard people. If I went all jerk-mode, that would just compound the negatives of the overall lizard-people environment, and the consequent drop in team effectiveness might also increase risk to the livelihoods of the people who can't leave.
At the end of the day though, his obsession with quantifying dev performance is doomed to failure. Goodharts Law can not be routed around by adding more metrics to capture all eng considerations. This can only work in a sanitized environment with low expectations and homogenous feature requests. In fast growing tech companies, there is far too much chaos and change for anything like this to approach an objective measure of what activity is producing value.
What I’ve learned over my decades a TL, EM, Director and CTO is that team performance has a human chemistry aspect to it where you need certain traits present, including a healthy dose of intrinsic motivation to do good work, and enough judgement on what that means in relationship to both short and long term business needs. Ultimately as a manager your only option to scale is to empower individuals as much as possible and relentlessly pursue and elevate those with the judgement and maturity to have more authority. Yes, it’s subjective, but there’s no faster way to kill the motivation of your top people than to reduce the output of their contextual reasoning to a metric. You’ve killed the golden goose while pursuing your control fantasy.
> It can’t be that you want your org to run without numbers. And it can’t be that you eschew quantitative goals forever!
Metrics are used by experts to enhance their ability to reason about a system. You should use metrics, you should not rely on metrics. Metrics don't mean anything onto themselves. Given the depth that the author is thinking through problems with them it seems very likely that he is using metrics in this manner. But once that intuition of how the team is doing has been accomplished, they have done their job.
The issue in this article is less that it describes a good system, but more that without those actual insights and ability to tweak them as Goodhart comes into play, giving out this system in the form of this article is harmful to anyone who tries to use it who isn't an expert already. Anyone who does not understand how to manage a team and build software who is using metrics to decide what to do without attempting to learn the underlying dynamics is focusing on the wrong thing.
Side note: I'm really curious how often he puts out tasks with an essential complexity of 0 with a high accidental complexity. I'd imagine in any decent team that should be a decent number of tasks. In the ones that track metrics sloppily, a majority of them.
> In fact, when discussing velocity metrics in 1:1 conversations, you should explicitly avoid comparing them against one another
These sentences are in 2 subsequent paragraphs. Compare people against each other, but you should explicitly avoid comparing them against one another. Sure thing.
> Ultimately, using lines of code, commit counts, or pull requests, to quantify engineering performance is a fool’s errand. All it does is create a culture where dishonest people write bad code to get ahead, and those with integrity are penalized and eventually quit in frustration.
But somehow velocity, which is an abstract, imprecise, and very error-prone measurement should be used as a primary metric for both team- and individual-based performance?
This article is such a mixed bag. There are great points like "don't estimate time", but then the author throws things like the quotes above. It feels like one of these "you're so close to getting it right" situations.
Also, as another commenter here mentioned – this can be gamified, because 5-point complex new feature is surely more time consuming and difficult than 5x1-point cosmetic changes. Hence, everyone should aim to take as many small tasks as possible in order to have higher individual velocity than other team members.
Are the business analyst going to fix bugs when the stories are wrong ?
The only real metric that matters is is the customer happy with the progress. It’s a team sport, and focusing on the devs alone is the wrong approach.
This assessment even ignores the other strategies a developer might follow like "minimizing regret". Failing a 1 point effort in 1h has essentially no consequences. Failing a 1-month 50 point effort ... do that twice and you're fired.
This rewards dumbed down, no-risk development. I've seen many teams work like this. Don't rock the boat, look busy, get promoted to management. Spend your time talking about looking busy with all the other ex-development managers who do the same.
Managers always come up with the next system that doesn't require them being the most knowledgeable person when it comes to what they're doing. I've never seen it keep working a significant amount of time.
I'm kidding of course, but it's a fun thought.
It's actually about management by metrics. The kind of Taylorite misapplication of story points that is standard in the industry even though it's repeatedly shown to clearly not work.
Is talking to the engineers an option here? This scenario sounds like a symptom of dysfunction. How would this happen with neither party talking about it with the other? What 'management' is being applied here if this can happen without being detected immediately?
This whole article seems to give a lot of credence to story points. And story points are a good idea. But they aren't a silver bullet for managing a software team. In fact, they aren't even a bullet. Talk to the engineer, ask what they are going to do. Maybe ask some questions about the bits that they don't seem to have thought about too hard.
Since there is nothing else that can be done, doing that must be enough. The story points will keep that process organised.
So it's not like we as a field is very mature. We might think we are special.
Since management has no idea, because good management is scarcere than good Software developers, they so to its labour force that they would do to another labour force.
They measure output without validating outcome. Even worse they still belive more output equals better outcome. Put in another way - more people churns out more. Industrial thinking. Doesn't work in software.
So this is not a software development issue. Its an organizational one, with its underlying communication structures being very much a problem.
We have all heard how well 20 people can outperform 200. Or 2000. Or a whole company.
It's Fred Brooks 101.
However, the relationship is a bit more of a 2-way street than the author admits to. The mice in the maze are studying the scientist too.
Meaning, such a manager would have a difficult time managing devs that know what tricks the manager is using, since the metrics are now targets.
Sandbagging estimates, producing verbose and pretty code, arguing against requirements, playing up requirements, etc. There are a lot of ways to make the metrics/targets play into the favor of the dev.
Though a good starting place, the article spends too little time on the most important part of the relationship: Trust. Devs and managers are people. If you go to the mat for your people, and they know it, then they'll most likely go to the mat for you too. No need to hide behind these metrics/targets.
People are not CI jobs. You have to listen. Pay attention. Your job isn’t to make sure they’re productive by putting together these bullshit metrics. It’s to make them more productive by removing blockers and being a meat shield for misguided stuff like this.
Granted that metrics are dangerous, this seemed like a reasonable stab at solving some of the problems (assuming it was done carefully).
Of course, it's no substitute for hiring the right people and having a less anxious upper management chain.
I'd argue the preceding thing you described is real software engineering. Decomposing hard ideas into a lot of super boring practical things with routine solutions == ???.
Also how do you quantify tasks? If one dev hammers out 20 widgets in a day is that comparable to another dev who spent all day hunting down a bug or coming up with an optimization that improved the performance of the app?
Yes! This is the crux of the problem. If a hard idea CAN be decomposed, or even a non-novel but complex need, it's the decomposition that's the hard work. It's "how to draw an owl" all over again.
E.g. you could build this new form field for 2 story points, but you could put in 4 story points, to make it a 1 story point task in the future.
> And conversely, if the team is picking overly complicated tech stacks, over-engineered designs, piling on tons of tech debt, and rubber-stamping all pull-requests, we should expect to see the velocity drop. And that will not happen if we score tasks based on their accidental complexity.
As much as we (as developers) may not like it, our performance is (sometimes) going to be measured, and it is often going to be done under the guise complexity points, and this part of the argument does seem at least like an improvement to the way it is sometimes conducted.
Why do only engineers need to be tracked and measured when reality is that systemic issues caused by managers have more of a negative impact?
Wonder where I can get some story points that aren't based on subjective feelings.
Bottom line: 10 years of non-coding experience in the industry (Sun & Intel), then about 5 years experience (assuming) development/coding and he's now a "serial entrepreneur" offering best practices advice for code reviews, architectural reviews... no mention of any product, technology or coding (where's the public Github account with demonstrable experience?) that he's actually delivered or maintained. I see a guy with 15 years of experience that got lucky with his options and is blathering on to boost his LinkedIn profile.
One thing for certain, he titled his article as something provocative to get the click-bait going and using "Moneyball" certainly caught my eye and made me curious. But after reading this article, it's fraught with problems and demonstrates the lack of experience and using very limited anecdotal evidence to apply to a field that's complex and not a single article solution, which I have to say, with less than 5 years of coding and development experience, he shouldn't be considered an "expert" that gives advice on this process -- he should still be listening to people and understanding how the SDLC process works because he's clearly missed that course at Standford -- He clearly didn't pay attention to Steven Blank...
Look, I know people with 20,30 (including me) years of experience in this field that would read this article and would point out, what you're reading is not only what inexperience is, but also what someone who's been too involved in the money and investment side of the business sees. If I had a manager in my team present these ideas to me and how they wanted to operate with 1:1's for example, I'd get them out of the company because they would be creating a culture of pointing fingers.
Also, as a side note, when I saw the title, I immediately assumed "outsourcing" article because the premise of Moneyball (Bill Bean / Bill James theory) is to use cheaper labor to get better results by doing things that can still get performance and wins without having to play the game the way in which the rest of the teams were, completely undermining the norms and trying to get better performance with less. This is a poorly titled article, the premise has nothing to do with the Bill James theory.
Lastly, I don't want to be totally negative,he had one good point in the article. Using time estimates for story pointing is a bad thing. His example of measuring complexity was a bit too vague and nuanced, but it was a better example of estimating to get to a valid burn down. But a broken clock is correct twice a day...