Don't measure developers, measure projects
anaxi.com
anaxi.com
One commit, one PR, might prove more valuable for your business than hundreds of other tickets closed.
Remove the KPIs and talk to your team members in 1-on-1s regularly. Give responsibility and trust to your team, and they'll be self-empowering.
Add KPIs and you'll take away their responsibility. Trust will be lost. Probability for innovative solutions will go down, your best developers will leave.
It is all about trust and responsibility. Hire for team spirit, conscientiousness, communicative skills and diversity.
If you can delete code and retain the same level of functionality (for example, because you found a library that will automate much of the task), then that's a huge win. Every piece of code that the team writes is one more thing that has to be unit tested, integration tested, documented, and maintained in the face of future changes. If they can offload that work to a library, or find a way to accomplish the same task by using existing code or by reconfiguring the application, that's almost always preferable to writing new code to accomplish a particular business need.
Of course, one can go too far in the other direction. It's easy to fetishize using external libraries for everything (like we see so often in the JS ecosystem). But that's where I would argue that the analogy of treating code like money shines. Sometimes it is worth spending money to have something custom made for your business, rather than buying an off-the-shelf component. Similarly, sometimes it is worth writing code to do something custom, rather than integrating a third party library, or reconfiguring an existing application. But that choice should be consciously made. Rewarding developers for lines of code written moves that choice out of the realm of conscious decision-making and into the realm of reflex. They get paid for more lines of code, so they're always going to reach for the solution that results in more lines of code, regardless of whether that code is necessary or appropriate.
Unfortunately so much development is "build it and they will come" rather than analysing and improving a system or solving a specific issue - which may or may not involve coding.
I worked at one company where a level 2 Engineer was bucking for promotion and spent almost a year (wall time) and ten plus man years redeveloping a system in Oracle that was functionality but added no real value.
And yes, I've seen this same behavior as well, at other large firms. The best way to get a promotion is often to launch something, so people launch, and don't worry about whether the thing they've launched adds any new functionality, or is yet another reimplementation of an existing framework that will make some junior developer miserable.
If someone produces an unusually high amount of code, that might indicate that they are producing a lot of functionality, but it might also indicate that they are churning out mounds of crappy boilerplate that should be abstracted better.
If someone is producing an unusually small amount of code, that might indicate that they are very unproductive, but it might also indicate that they are very efficient and are finding simpler ways to express functionality.
All of these things are things that you want to be aware of. The mistake is treating metrics like this as something that is correlated directly with performance instead of just tools to help you understand the bigger picture.
I don’t agree that short is always better. Sometimes it’s better to write the code so that it’s easier to understand than to use smart tricks to make it short, because the next person looking at the code might not be on to your tricks. You can argue both ways on this of course, but I prefer clean and easily understandable because I’ve seen what happens when you don’t do that and a junior programmer has to fix something 15 years later.
I also disagree that libraries are better. You seem wary of JS, and that’s good, but no libraries are magical and none of them can be trusted to do something for you, unless you look under the hood.
Especially not open source libraries. If you don’t know what is going on inside the libraries you use, then you’re heading for a world of pain. Of course some libraries are better than others, but using them blindly is wrong. And if you have to read up on them, then they aren’t really quick fixes.
We had a developer who used a library to handle decimals. We have our own library for this, that we’ve build because almost none of the open source libraries for decimals are very good at handling memory allocation and floats and computers aren’t 0.00000000000 they are 0.00000001436 etc. But our developer didn’t know we had our own library for this, so he used a public one. One that uses around 10 other libraries, one of which didn’t handle memory allocation the way you’d want.
So suddenly our decimals were wrong, but the computer didn’t know they were wrong, because technically the library worked it just wasn’t specific enough. And it cost us a lot of money to clean things up afterwards.
This is the extreme example, but it’s never been more relevant than it is today. Especially not in a world where less and less programmers seem to actually know how computers or even compilers work.
I get why you may object, but open source libraries have failed us a lot and by contract the .net framework library hasn't.
There are a lot of great open source libraries, that are both well documented and maintained, but there are a lot more that aren't.
I guess you also addressed this problem, right?
Distributing knowledge inside the company is an issue, that's even more crucial in software engineering.
I wouldn't go that far. You use glibc, right? Have you looked under its hood? If you're writing Java, have you really looked inside Spring or Hibernate? In the .Net world, how many people have really looked at the ASP.Net source code? How many Python developers look at the code for Requests? Even in Javascript, I can't remember the last time I had to crack open jQuery and take a look at its internals.
I'm not sure he is drawing the line in the right place. Business value requires functionality which requires code. Business value would be an asset for obvious reasons. Functionality and code would be liabilities. Functionality and code that doesn't provide business value is waste. You want to minimize both functionality and code. Maybe it is just the analogy breaking down though.
I don't think business value necessarily requires functionality. Business value is generated by code operating inside a specific business process. Only part of that process is software; the rest of it exists in the minds and the actions of the people interacting with the software. In some cases, it's easier to change the business process than it is to change the software.
Obligatory: https://www.folklore.org/StoryView.py?story=Negative_2000_Li...
Juniors tend to be overwhelmingly concerned with basic literacy. First its a "does it work" and then it graduates to "how to write the code". This often comprises thinking about tools, frameworks, dependencies, and code style. These aren't deep problems that moves a business forward. These are competency problems that keeps a person employed. This is a distraction that consumes developers with busy work instead of thinking and planning about how to make a product better or how to release faster.
Worse still, too much emphasis on the particulars around the how of it can blow out a 30 minute task to a 2-5 day task with all the unnecessary scaffolding and extra process. This is particularly frustrating when realizing these concerns can be immediately eliminated when the proper automation is in place. It is possible to really get good at spinning on mouse wheels and cranking out those stories a little bit faster, but it still isn't moving the business forward. Nothing forces out strong developers faster than cliches like: "That is the framework way" or "You're reinventing a wheel".
Some better metrics to really measure real world success:
1. does the change solve a business problem
2. does it pass the project automation
3. does it reduce total code instructions
4. does it close out defects
5. reduced execution time
6. improvements to project automation (faster developers)
7. improvements to documentation
Reading code is mostly way harder than writing it, especially if you read code from someone else. Optimizing for readability is the game changer for maintainability.
* Use expressive function and variable names.
* Have clear consistent structure across your project(s).
* Use simple code where possible.
* Do not write one-liners that do multiple transformations. Each line should have one purpose.
* Document design decisions in code. Don't document "what" your code does, that should be readable by itself, but why you made the decision to do it like that.
* Documentation library style comments for complicated functions helps a ton to know what the function is supposed to do.
* Add tests that specify the expected behavior. This not only helps when changing code, but also when reading.
It's not readable code XOR documentation, it's both. Readable code is the "what", documentation is the "why". Even if the docs get outdated, the why can still be insightful.
Non of the above suggestions usually reduces code, but it will help everyone else, including future-you in 6-months, to work with your code.
Using abstractions, removing dead code, refactoring to share common code paths, etc, all reduce the amount of code you have, and all typically increase the readability of your code.
This doesn't seem at odds with your bullet list for maintainable code.
A useful maxim is: working software is an asset, source code is a liability. More code means more maintenance cost, so accomplishing business outcomes with less code is good, to a point.
The total instruction count is the number of instructions from the source code that are actually executed. This isn't the lines of code that people read. For example a loop may comprise 100 iterations and 3 lines of code. That loop will likely result in 300 instructions. If a loop contains a child loop than the instruction count is something like:
(iterations * overhead) * (iterations * overhead)
I am also not a fan of pronouns, which in some languages is the this keyword. If I have to guess what a reference is contextually I am wasting time that could be better spent making a code change.
Does cramming 5 lines of code into 1 line make it less code? No.
Reading code is harder, so the less code there is to read, the better.
Sure if you can reduce code without making it more complex, that's a good thing.
Measure outcomes. Only outcomes. Try and make sure your metrics capture the real outcomes and not just what you see in a lab. This usually means you need to use field data to get that information.
That may not sound like a very productive environment, but that is the environmental in which most of the world’s software is written. For those engineering managers that have to operate in it, measuring your ability to execute is the best you can do. You might as well do it correctly (which is hard).
And if you’re one of the lucky few working in a less dystopian environment, don’t be too quick to gloat. Enjoy it while it lasts, you’re just one acquisition or reorganization away from joining the rest of us in corporate mediocrity.
(Full disclaimer: I am also currently working as one of those project managers you mentioned. If your project managers aren’t communicating clearly about how engineering impacts outcomes, find new project managers.)
You appear to be one of the lucky few not in this situation, but you don’t have enough experience to realize how lucky you are - or how quickly your situation could change. You think everyone is happy with their organization at Apple? Think again. Like I said, you’re just one reorg away.
Please refer to the last paragraph of my previous comment, it is extremely relevant to you.
Managing your team is table stakes. Managing 360 isn’t a sign of lack of experience.
Perhaps you should just stop trying to give me unsolicited career advice, and actually address the comment you’re replying to? Just a thought.
He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? You can't mean the final shipping SW because that's usually way too late. You want to figure out if your velocity is going to let you hit your ship date well in advance of the actual ship date so that you have time to deploy process fixes/hire more people so that you do hit the ship date.
Evaluating teams on outcomes will force them to collaborate to make something great. Evaluating teams on collaboration will just make teams do whatever it is you’re measuring.
None of the proposed metrics capture project health. Test coverage is the closest and that still isn’t great. (Unit tests are important but you can have 100% test coverage and a 0% working product. In fact, this is how every project begins before the first line of code is written.)
No one gives a damn how many pull requests happen or how many days apart they are.
1) Does the software work yet?
1a) How much of it works?
2) Does it do what we want?
2a) .... everytime? Less? How many 9s?
Report on the things that matter. Track the things that matter. Everything else is actively harmful and distracts your teams from doing what matters.
Also, when you ship the software you haven’t finished, you’ve begun. It is absolutely still useful to keep metrics on outcomes and track improving them over time. It is absolutely not “too late” and any planning that assumes software is done after you ship it is flawed. Features can become done, software never is.
For example, your velocity has changed. Why has it changed? Are you bringing new people up to speed? Have you suddenly hit some unexpected complexity? Your mean time to resolve issues has gone down. Are you releasing code with simple bugs you should have caught? Have your team started picking them up sooner?
Whenever something changes there's questions you can ask, the answers to which might well help you do better. Using metrics as an early indicator of change can help.
2) That's an argument for literally any possible metric. You could make an equally passionate argument that looking at and keeping track of free/busy times on calendars or soda consumption or how full the garbage cans are on the floor as an indication and early warning sign that something changed. It doesn't mean those are good things to track. Pay attention to the things that matter. Align whole teams and organizations around those things. Don't let them get distracted with metrics that don't matter.
3) Metrics are a double edged sword and influence behavior. If you're tracking the wrong things, you will get the wrong things.
Funny is that as project leader one should click through project as in using it to see if it is working instead of wasting time on finding proxies. If leader/manager does not know how to use system or never clicks through then he is useless...
If what you have to remember to make a certain change means you have to keep 27 things in your head at once to avoid making bugs, you might want to consider re-organizing your code so this number is smaller.
https://www.amazon.com/gp/product/0932633366
It talks about the different between informative and motivational metrics - the former being just for awareness while the latter being intended as explicit targets for employees. While it's easy to bleed the former into the latter, there's nothing inherently wrong with measuring game-able statistics as long as the incentive to muck with them is minimized. Easier said than done, but not impossible.
The article is pretty dismissive of story counting, which is a semi-popular approach to estimation:
https://martinfowler.com/bliki/StoryCounting.html
(Anecdotally, we had someone look at our teams' previous sprints and found that # stories / sprint was more consistent than points per sprint over the time frame they reviewed. YMMV.)
The focus on velocity-related metrics seems smart - Git Prime did some analysis on this and found that frequency of commits was a useful measure:
https://blog.gitprime.com/prolific-engineers-take-small-bite...
I don't see how these are any less game-able than klocs though.
Concerning Gitprime, I completely agree with you. And they do say that the point is to game their metrics: https://help.gitprime.com/general/cant-you-just-game-the-met...
* release often, release early, each release should function though with less features.
* break large goal to small steps, one step at a time.
* sprint every 2 weeks or 4 weeks.
* 15-min stand-up every other day
* etc,etc
In my experience, if business stakeholders are seeing real value shipped to customers every week (or 2 if you must) then they don't get very antsy. Especially so if they are using a release-early, release-often approach to the product, where they see a project as an opportunity to explore what customers really want.
The problem starts to come in when the feedback loops are long. If they ask for something in January and don't see results in June, then they've got 6 months to wig out. And one of the ways they'll try to manage their fear is by getting up in developers' business by imposing all sorts of metrics.
embedded product is notoriously hard to manage and I found Agile process actually helped quite a bit, via Scrum's 'story points' method.
Scrum (what you mostly describe) does not address this, really. Except that it attempts to reduce metrics down to a primary metric (there are others): Story Points. These are the primary concern of a Scrum team, and are developed by the team members as an estimate of the size of the stories (tasks) as they see them. These are not given by anyone else. These are not used by anyone else. The team then evaluates themselves (if they do this correctly) by constructing a control chart [0] to see how they're performing. If they trend down, they identify the cause and share the reason with other teams so that others with a similar cause can address it more readily. If they trend up, they identify the cause and share so that others can attempt to duplicate the success.
Scrum does not address how to do personnel performance evaluations so it doesn't stop management from trying to use metrics for the purposes described in this article. For example, story points (if used as an evaluation metric for personnel) become useless as people will just inflate their numbers in a continual game of oneupmanship. At which point a large amount of the value of Scrum is lost.
The Lean school of Agile is a philosophical descendant of Deming's work in Japan post WWII and the Toyota Production System (called Lean as it spread). This school of thought does address the issue discussed in the article in that Deming's views have you reducing your focus on individual performance and increasing focus on performance of teams and projects (title of the article is related to this). Deming and others recognize that the system in which work is performed effects the outcome more than the workers themselves (NB: His focus was largely on manufacturing, but this works in service industries and to some extent in our discipline as well). Since the view in that school is on the system, it does address the problems described in the article (if you get management to understand it). So by focusing on the project level (really you should probably go higher, department or even enterprise level) you can understand where problems are occurring, where there is waste worth cutting or addressing, etc.
That said, I do not recall seeing some of those points I made (gathered from Deming and other sources) in any of the books I've read on Lean as it relates to software. It could just be that I have forgotten or missed that information (may have glossed over it as I had already internalized it) or was in sources that I haven't read yet.
I'm sorry, but if you think "story points" is something useful, your organization is already doomed.
Story points is an inflationary currency. When your developers realize that somebody important in the company cares about said currency, they will make sure that their work has more of these "story points" in "sprint planning".
There's a negative Pavlovian reaction developers have every sprint when 1) "commitment" was not achieved, or 2) story points decreased. To avoid that pain, developers will pad points. This means that when things go according to plan, large sections of a sprint will have no work or people will try to appear busy.
The easiest way to avoid that gamesmanship is to not measure things that don't matter, which is all inputs. What matters is outcomes, which can be in the form of money, usage, satisfaction, etc.
You do need some time now and then for refactoring and that's how we did it. Not everything needs to be formally estimated and scheduled.
Story points are supposed be a tool to help management with realistic scheduling, not a way to push developers to work as fast as possible. If there's a lack of trust between engineering and management, it sounds like you have bigger problems.
There is no empirical evidence that story points are more effective than other forms of estimation. (There's also no evidence that any form of work estimate is related to business value, which is the original criticism of measuring inputs).
Secondly, the perverse incentive around sprint boundaries exists and can create bad management practices. If management is actually looking at those estimates retroactively, it means that the team is being evaluated on the performance of their estimates, not on the timely delivery of value. If management is not looking at them--either in the long term or the short term at a sprint boundary--why bother in the first place?
Additionally, if it was routine for a team you were on to finish ahead of schedule, my guess is your teammates and/or manager was padding the estimates. In which case, we're back to what I consider a situation where both the company and employees lose because of disengagement.
Story points are a way for your teams to improve themselves. If they don't understand that, then teach them. And never report the points (directly) to anyone above the project lead or perhaps one level above that. Any higher and they are too disconnected from both the team and the work to understand what they're seeing.
The only thing that higher ups should care about are trends up and down. Your team, after say 6 months, should be pretty steady in their performance if they're using story points correctly. Downward trends should be identified by them and their project lead early and addressed, and if it's an issue that may impact other teams they should communicate that up and out. Upward trends should be celebrated but also examined to see if some change could be duplicated elsewhere (an improvement to your build configuration resulting in faster feedback, adoption of a new development tool or technique, etc.).
These points have nothing to do with actual value produced to the business, or the developer's worth / skill. It's also very easy for a developer to begin to inflate the estimation of tickets they believe they will work on, creating a false sense of confidence.
Alternatively, if people work at a "shop" where tickets are so fungible that people pick up whatever the top ticket on the board is, it tells me that they are working on low-value, repeatable things where expertise is unnecessary.
It's also not clear that increasing story points (even if they were somehow related to work / hours perfectly) would be useful. I've seen teams drop down to do minimal "work" for months and produce tremendous value because they were reorienting the product to do things successfully. An organization that looked at that labor input, sans objective outcomes in the product, would be doing themselves a disservice.
Story points, tickets closed, lines of code: ultimately all of these are a belief in the labor theory of value. But that's the thing. I don't want labor, I want value, and the tragedy that is input-focused measurement is just due to it's easier to do than measure product outcomes. It's always more convenient to say "We did 24 story points this week" than "Our users hate our software, and here's how we know it."
This is true.
Story points don't translate to value. They can't. A nail could be worth 5 cents, or it could be a million dollars. Value is determined by the customer and their need.
Story points translate to cost. Customers determine the value. They examine the cost. And they determine whether the value is worth the cost. Is that 5 cent nail worth 5 cents or is it the nail that'll save the project?
Story points are about developing an effective and consistent estimation tool (how much will this cost or how much time will this take to produce). If your team is inflating their story points, they're idiots.
Assign a number to a story or task. If the team has developed consistent and reliable estimates (this takes time to happen) then this number translates with some reasonable fidelity to cost and time to complete the task. If the numbers are unreliable indicators of cost and time, then the team needs to examine their estimation methods. Perhaps bring in other people with more experience to help them.
The team (themselves, no one else) has a history of completing so many story points per cycle (sprint, week, month, whatever). So they know that if the customer has asked for an estimated (their estimate) 50 units of work and they can only complete 20 per month on average, then it'll take 2.5 months on average to complete the work. This lets them, you know, plan. And discuss with their customer the priority and objectives. If the customer is only paying for one month of labor, the customer will have to reexamine their objectives and priorities.
The customer never sees the story points. They see the cost and time estimates the story points turn into.
The managers never see the story points. They see the cost of the projects and the income from customers.
The only people who should see and care about the story points are the developers and the project lead. Maybe a statistician if they can't do the math themselves.
There is absolutely no evidence that story points actually correspond to cost in a product organization (as opposed to a contracting billable hours situation), because the long-term scope of a project is unknown. When does a project actually "end" with respect to its cost vs value production? Is this accounted for via story points? Are all of those things known when starting the first iterations? I would submit the answers to these are "we have no clue."
Something that does however correspond to cost is well, cost. Team's have a known salary burn rate, and this is completely independent of how many imaginary points they make. Any organization whose value output is being measured where two weeks is actually such a large timeframe that it needs to be broken down into more granular estimates is 1) delusional about the predictability of their actual value output and 2) pipe fitters working on easy, well-known features.
There are many ways of estimating projects as a whole that are equally as (un-)reliable for estimation and take way less time, and these can be coupled together with financial rationalization from the business. If two week increments can't have significant variance for the business to make a profit, and this business is in the extraordinarily high margin software industry, they're working on the wrong problems.