Measuring an engineering organization
lethain.com
lethain.com
"Management" needs to stop thinking that software is an assembly line and that they are managing "workers". The actual workers are CPUs and GPUs - and coders "manage" those - coders are the new managers.
The bullshit jobs people worry about are those managers who think they are arranging and orchestrating coders and if they shuffle them faster they will get better results.
As more and more of the real world becomes eaten by software, measure that.
I like the comparison to a engineer in a factory.
In 1960 how did we measure the impact of an Detroit factory engineer? "Did the time to turn around the factory line drop this quarter?" That is a high level, impactful metric that has many inputs and would be a "managerial or executive level" metric with bonus attached
Today that same metric works for "has the turnaround time for a new release of the data pipeline improved this quarter"
Those are the metrics that will matter. Measure those.
Visits, clicks, read time.
> Novel writers?
Sales, reviews.
> Pop song composers?
Hits.
> "Management" needs to stop thinking that software is an assembly line and that they are managing "workers"
I can't avoid to find funny the thought of telling management, as an engineer, what they should do.
---
Good software engineers should embrace good productivity metrics (like the ones presented) as a meritocratic path to success.
I find there are 2 distinct set of metrics for engineers: ability to delivery, and to create value. Both important, both difficult to measure, both possible to approximate, and both benefit the business.
Metrics don't exist in a vacuum - they not only tell us where we are, they set guides to directions we want to take.
Also required quote:
"Some one asked me " Bert Bacharach, you have written so many hit songs, what is the secret to writing a hit pop song. I said If I knew that why would I do anything other than write hit pop songs?"
The fact we measure this way has destroyed journalism as a profession that writes important and accurate stories that inform the public.
It also reveals the problem that a locally optimizing for something can lead to reduced global levels of that thing. E.g. a journalist on a site making click bait might increase the views on his article while also turning people off the site as a whole, destroying it.
> Visits, clicks, read time.
> > Novel writers?
> Sales, reviews.
> > Pop song composers?
> Hits.
These are all proxies for revenue. So, you could just measure revenue.
Then you fall into the trap of producing mediocre creative work to try to maximize revenue.
It might be satisfying to deliver a hyper-optimized end product, using optimal algorithms everywhere, with everything tuned so that it delivers the maximum performance with the minimum resource usage. But is that what your customer is paying for? Or are they paying for a website that works well, and is operational at the end of the month?
Not everything has to be a work of high art. Pop art is fine too.
Like rating a 5-star chef on individual eggs cracked per week rather than gourmet distributed systems delivered and Paxos hungers sated.
And every single one of those eggs gets evaluated independently - because as a 5-star chef, you're only as good as the last meal you made. A bad review is not a good life event to experience.
If you want to get rated on "got some shit cooked", you might want to try working at the griddle in a local diner instead. (Although... throughput still matters)
Think of it this way - it's fine to "tell your elected representative what to do" - perhaps we should look at executives as an elected function
There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach.
Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity.
A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com
How about nothing? Honest question. I am sick and tired of being treated like cattle. I've never seen a single convincing argument that any metric of my outputs has mapped to some productivity metric in a meaningful way, but I have absolutely seen them weaponized against me and others to punish and fire engineers. All of these metrics have only ever boiled down to a negative reinforcement mechanism, in my experience.
In an organization with hundreds or thousands of developers, there will be people either lying about how productive they are or genuinely think they are performing above average when they are not. It's like how 80% of people think they're above average drivers.
On the other hand, you may have excellent developers that are overly modest or not loud enough to sell themselves. They are truly exceptional and above the curve and should be rewarded for that. Some of the 20% of drivers that don't think they are above average may actually be above average.
Objective, measurable signals help to find the outliers at the ends of the curve that may otherwise be missed.
This is, so far as I can tell, ~5% of them.
And as much as many of us on HN are developers and like to toot our own horn, some of us suck and don't get better with time. I mean in small organizations this is typically easy to figure out, but in larger organizations it's a problem that can persist for long periods of time.
The same way we deal with good developers? With managers, of course. That's the whole point of their role. Every developer productivity metric I've seen speaks to some disparity between engineering and the C-suite. I don't understand where these products originate that seek to do the manager's job for them but worse. I've been in management roles before, the numbers don't and can't always tell you the full story, no matter the size of the team or the quality of the data.
As always, good people are hard to find.
Ask the developers that work with them? Who likes working with someone that isn't pulling their own weight, or worse, that is just making your own work harder?
And asking good developers to identify good developers doesn't work either. Good developers hates working with bad developers, true, but like all other humans they hate working with all sorts of people for different reason so they will point out a lot of good developers as well.
Instead what works is if you have lots of data from diverse group of good developers. That way the personal noise gets cancelled out and you are mostly left with the "I don't like bad developers" signal. This isn't trivial data to get though, especially since people usually choose to work with people like themselves so a team likely isn't diverse enough to get you this kind of data, and people from other teams aren't familiar enough to give good data either.
3 of them point at 2 of them and say they are bad.
You fire 2 developers.
Nothing gets done. Turns out the 2 were the good developers.
Congratulations, you succeeded at making another bad metric.
If a bad manager keeps only bad developers around it will kill a (software) company just as fast as anything else. It turns out that paying customers like their software to work and be free of data corruption.
The people that need to interpret the metrics might be a professional that understand their meaning, or they might be an AI following arbitrary instructions. The first group will use the metrics to create a better team, and the second will make people angry.
https://www.cbc.ca/news/business/sony-cyberpunk-2077-playsta...
Edit with more arguments:
You could of course blame managers for that, but delaying a game since your bad developers aren't capable of delivering isn't really an option either. How many years will it take those developers to get things right? What they would need is to delay the game and replace the developers with good developers, or scale back the complexity of the game to a level those developers can handle.
The company in question choose the latter, they said they will use unreal engine for future games instead of an inhouse engine. So they admitted their developers aren't good enough.
Build vs. buy is pretty much never a question of developer quality, but a question of ROI. Even with the best devs in the world, building something on par with Unreal (or other commercial engines) is a huge outlay of capital. With relatively little return over a single game, you need multiple hits to pay off the engine.
There aren't a lot of companies that can afford that.
Here's a decent write-up of some of the major issues that actually plagued that project, as opposed to "bad devs": https://blog.devgenius.io/cyberpunk-2077-through-the-lens-of...
All of those issues were there for their last project as well, but somehow they only failed this time even though they had a much larger budget and larger team with more developers. The most reasonable explanation is "bad devs", even though that will never be an official reason. When you swell the team size quality tend to take a hit, and when the old good devs becomes managers and the new devs try to replicate the work you often get problems.
That article describes how you need to work if you have bad developers. CDP-R assumed that they still had as good developers as they had previously, which wasn't true and that bit them.
All the indicators point to "scale". Having worked in games for almost two decades and having seen anything from solo dev to teams of hundreds, and the growth in between, my experience leads me to say "scale" as well.
Do you have any indicators for the "bad devs" problem? So far, you're just stating it as if it were categorical truth.
Whenever management makes a new attempt at getting more precise estimates I always tell them the only way to get things done is by doing them. So the number one goal should be to remove obstacles, simplify processes and make sure tools work to our benefit. For some reason nothing much ever changes and the question for better estimates returns. It’s pretty weird.
If the company is only trying to measure developer productivity, they're missing the big picture. Developers will only get blamed when the number doesn't match some hidden expectation.
Good organizations measure a lot of things, such as tech support effort (mentioned in this article), roadmap churn, number of unexpected feature demands that derail forecasts, and other metrics that evaluate the company rather than laying blame on a single group.
Measuring roadmap churn and the impact of unexpected feature requests has been very helpful for me at metrics-heavy companies. When management starts asking why things haven't been shipped yet, it's amazing to pull out a quantified list of all of their unplanned requests, roadmap changes, and other management issues that set everyone back.
As another example I like using story points/velocity to get a feel for the team, but I don't think you can extrapolate on that data and expect the results to be valid. I think most of the frustration from devs come from the measurements being used to try to plan out the future when that isn't really possible. IMO you can choose a set of features OR a release date but definitely not both.
I'm no expert but I've read a fair amount on different strategies to try and make timelines for software and my big take-away at this point is "don't make timelines for software. If you _have_ to, then keep it probabalistic and don't promise anything by an exact date". I think we as devs need to kinda educate the rest of the business that it's a tradeoff between features and release dates. I like the "iron triangle" model but you can't just add more people to a team and expect things to be done faster (Mythical Man-Month), so it's really just scope VS release date.
[1]: https://wakatime.com
OSHA would disagree, and even racetracks have rules.
I think a lot of developers on HN just have a problem dealing with the fact the rules are going to be applied to them. If you become an owner of a company your engineering rules go away, but then you have all the regulatory fun of a business owner trying to make a living and selling things at the same time. Working for a larger organization tends to have benefits like steady pay, but in return they ask for you to justify your expense and it's very possible the people asking for justification have no idea exactly what you do.
I don't have a problem with rules being applied to me, I just see it as dysfunction that any organization gets to the point that they need a significant set of rules. The more rules, the worse the dysfunction, typically. Exceptions include life-critical applications, where the entire existence of the company hinges on stripping human error from the product. But even then, more rules ≠ less problems.
This even works introspectively; it's a bit like fitness tracking. Is a little bit of data helpful? Yes. But who is out there reading into every time they got sick and missed a few thousand steps, or thinking about every utterance and flatulence during sleep and how it signals an overall moral failing needing to be addressed? People with mental disorders, that's who.
In the short-term I think most people would hate this, but then I think we need to choose between raising the limit or following it. We all make the laws so we can raise the limits, and this approach completely removes the potential for abuse or playing favorites. As an added benefit it wipes out a lot of what cops spend their time on, so if you don't like cops we could justify having way less, or if you do like them then it frees up their time to focus on more important things.
I also recognize this opens up a huge potential for abuse by the govt having all this data, but we already have it for tolls so we might as well use it for good - having human police officers handle stuff like this is a huge risk for abuse and a waste of time.
How democratic is your country? Will the limits raise or will your government take the extra revenue and keep things unreasonable?
You will find that people have many different answers for those.
This problem definitely resonates with me, and I have some very primitive ideas of tracking (i.e. tagging tickets that fall into these buckets), but I would love to learn from your experience if you're willing to share!
Problem of many organizations is they are progressing without a vision and have no idea when progress gets blocked. Since nobody really measures blockage, workers typically drop the thing that got blocked and go chase metrics on something else and don't tell anyone.
https://www.notion.so/coatue-external/Scaling-Up-Engineering...
Instead measure ratio of outgoing requests to incoming request if this ratio is greater than one or approaching one than you will not be able to execute with your current plan since you are not able to dispose issues in time. Your team is in danger zone and there is a risk of burnout of your team and non delivery of the project. This is also useful because you can go to the management team and tell them you need additional headcount and present this data and you are likely to get your request approved since it is backed by data.
Same thing goes for other metrics as well. Your metrics should be related to your goals they should clearly tell you if you can achieve your goal and how near/far you are from it.
Last but not the least don't forget about human touch since you are managing humans. Metrics should just be the starting point to dig deeper to solve the organizational issue they should not be used as final goal.
What kinds of metrics are used for planning and operations? Productivity sinks?
I view engineering teams as a collection of highly driven professionals whos productivity levels are by default at 80% max daily.
Their productivity is impacted by bad planning, bad developer experience, bad documentation, meetings without agendas, 1:1s with no support or actionable feedback, individual recognition instead of team recognition, etc.
It's the engineering manager's job to remove the things that impact team productivity, measure the impact, report the issues affecting productivity, and plan to make them not happen again in the future. These are external to the team, things soaking up their time and making them stressed.
When you try to measure an individuals productivity of any role, you are now impacting productivity and should be removed to allow them to do their best work.
Use surveys to gather feedback from teams and allow teams to weed out, or help to reduce, internal productivity blockers.
And please communicate this to all members of a team, it's one of the primary internal to the team stress sources I've experienced;
role != authority
Roles are a collection of responsibilities, they do not grant additional authority.
From my experience what I've mentioned above would certainly make my day to day alot less stressful.
- measure teams not individuals
- stick to measurements related to planning and operations
- implement one measurement at a time
I didn’t see it in the article (yes I skimmed) but I’d add that it’s beneficial to develop measurements in conjunction with engineering teams.
However, software engineering in a larger organization has a different goal -- it's team sport of building & operating software to serve the business purposes of that organization.
With this goal in mind, one's ability to build upon & maintain an inherited codebase is definitely relevant to success.
> The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.
> Managers tend to use measures as surrogates for strategy, acting as if measures were in fact the strategy when making optimization decisions. This appears to occur even if a measure-maximizing choice ultimately works against the strategy.
For those interested in history, there's a lot to learn about metric abuse from the Vietnam and Iraq Wars. For example:
> On October 27, 1967, The Wall Street Journal ran an un-bylined blurb from its Washington, D.C., bureau on the front page talking about a “victory index.”: U.S. strategists seek a “victory index” to measure progress in the Vietnam War. They want a single statistic reflecting enemy infiltration, casualties, recruiting, hamlets pacified, roads cleared. Top U.S. intelligence officials, in a recent secret huddle, couldn’t work out an index; they get orders to keep trying. (from: https://www.theatlantic.com/technology/archive/2017/10/the-c...)
Also, I would widen the perspective of "CTO needs to provide data to CEO" / engineering effectivness a bit, as this is not the most important thing you need to know. Ultimately, as a company you wanna know if your business is going in the right direction or not. You want to spend more money on things & activities that add to the value of your product offering in the eyes of a customer so that the customer will pay for it and you want to spend less money on things customers will not pay for. But what are the value-adding activities and which are non-value-adding? When will a customer buy your product or service?
For me, this sets a certain order of importance of what things to measure/quantify to answer the following questions:
1. Deliver customer value. Are we building the right product?
2. Generate business value. Can we generate revenue?
3. High value product. Do we keep quality high and build the right next features?
4. Code quality. Are we building the product right so it is maintainable and exensible?
5. Team chemistry. Is the team aligned on the goal and healthy in their interactions and spirit?
6. Process efficiency. Is success repeatable?
===
(1) Answering that should be not too hard as you can measure any kind of customer feedback on your products, be it youtube likes, alpha tester feedback, support calls, surveys.
(2) Once you know you have a product customers want and like, you need to know is the customer facing part of your organization (marketing, sales, training & education and support) connecting to said customer, so that they can sell and actually generate revenue?
This is much harder to get data for. You can measure the number of licenses, lost and new customers, or the trends in the business volume of services, but what does that tell you about the ability of your organization to be able to sell a good product to customers? I am not sure here.
(3) is to reflect on the business success (ROI) on new features that you implement. Do you keep building a high value product? Are we effective in communicating the new features value proposition to our customers? There is soo much to measure here.
Feature level: Measure via BI the business impact of new features, how often are they used? Measure the resources needed to deliver a feature. Measure number of support cases / bugs reported per feature. Measure the estimated time vs the taken time for a feature.
Quality of the overall product: Measure the yearly mean number of support cases per week. Measure the yearly mean number of bug reports per week. Measure the yearly mean number of crash reports per week. Measure number of major field incidences per year.
Do we make the customer feel cared about after the sale? Measure lead time to resolve support calls. Measure customer ratings of support calls.
(4) Code quality like compiler/sonar warnings or test coverage are the easy measures, but might not give you insight. I prefer to look more highlevel again at the product quality which ties into (3), like How often do tickets come back from testing to development? Measure number of regressions reported by customers after a release. Measure number of (real) hotfixes needed after a release.
Answering the more fundamental questions of code quality like "how easy is it to add new features" is a bit more difficult. I have not found good measures yet.
For (5) totally agree with the article, you should never measure how many lines of code were produced, how many tickets closed or the like. The question of team happiness is most important for team work. Ther are good tools for that like OfficeVibe or Glint.
Measure the employee turnover. Measure the number of uninterrupted hours of work time / total time present at work per week. Measure the mean Office Vibe score. Many further answers could be drawn off of Office vibe: Are people at ease, having a good time and enjoying interactions with their peers? Is there no sense that single individuals try to succeed in spite of the efforts of those around them? Was the work a joint product? Was everybody proud of its quality? Do they take enjoyment in their work? Is there trust and mutual esteem among the peers?
For (6) you want to know: Are our processes such that success are repeatable?
On a company level: Measure number of botched releases / roll backs. Measure number of failed audits per year. Measure number of open CAPAs per year.
Per Team level: How long are compile/ CICD times? How long do code reviews lie around before taken up? How easy is on-boarding of new employees? etc