A software engineering manager guide to measuring an engineer’s performance
jmoses.co
jmoses.co
what?
Please don't listen to this. Give junior engineers bugs right away. It forces them to set up their environment for debugging and begin to understand the flow of the project.
Even if he/she needs to be guided to the solution, this is a GREAT litmus test for the standards of your documentation, and ease of environment setup.
Where they proceed from there depends on their strengths and interests.
I'd actually lean the other way: No one should be adding new features to an existing codebase without having successfully done some fixes first.
Also jr. devs require lots of sr. dev time to verify their work. But bug fixes are usually easier to verify then features because the ratio of code written to investigation is much smaller.
Doing the actual code changes to fix a bug which has already been analyzed and localized to one or a few sites in the codebase is a bit more reasonable.
That's precisely why it's great for a starting point. It's work that mostly consists of research of the current code, to look for inconsistencies that cause a well-defined misbehavior. It involves writing little code and therefore involves little decision-making if any, so you avoid the newcomer making changes or adding stuff that, in ways, may be inconsistent with how things are done in the project.
Right - so no time like the present to get started understanding!
Have those expectations never clashed for you?
Right?!? That's how I get every junior developer into the code. Hell, that's how I get myself into the code at a new job.
It is also important to select right kinds of bugs for newcomers.
Certainly, guide them towards easy-ish bugs rather than complicated ones right away, because there is something to risk management of new people attacking gradually-increasingly critical code / difficult bugs. Of course you probably wouldn't give a random interviewee root access to a credit card processing system, but you should ratchet-up the level of trust incrementally as fast and as frequent as a staff member has proven themselves (treat people as adults, not kids).
Lol, some "managers"
There isn’t a better way to learn a codebase and new tech than one issue chosen by you at a time
The great majority of devs are not very good. (Same for managers.) Here at HN you are only seeing the pretty good to great devs. Not boot camp grads and unranked school grads.
The other thing is that this comes from the management school of management. One where managers mentor ICs and managers drive skill growth. As opposed to the SV ideal (rarely met) where mgmt gets out of the way. They remove blockers, not give opportunities. They communicate information, not hoard it.
So, from the management school of management style, managing lackluster devs (the 90% that you reject at phone screen), it’s true. These devs get in over their head when the first thing they have to do is actually understand the spaghetti codebase. This is not a road to success for these devs.
With that POV and that reality in mind, the advice is sound.
Ugh, not this again. Obviously don't hire people who aren't good at their job. But most improvements come from investing in your team.
> are they putting their code up for review [...]
Missed one: Are their code reviews/pull requests high quality? I.e. do they go out of their way to document how they tested it? Reproduction steps? Do they invest time in making code reviews as easy to review for other people as possible? Or does their code reviews always take multiple rounds of review due to sloppiness?
Very well said. Completely agree. Which is why I think the ability to "learn" as you go is so valuable. You don't want to hire a Know-it-all. You want to hire a Can-learn-it-all.
I agree on this but you have to cut your losses at some point. Some people just aren’t motivated. A surprise pay raise, catering lunches, taking them to an offsite, etc whatever motivational strategies you have may provide some short term result, and that could be sufficient if your goal is just X and it’s in the line of sight, but sometimes raising the bar on your team means finding a different spot for the unmotivated folks, and if they’re not investing even in that, then cut your losses and let them go.
Having personally burned myself down in such an environment, a good interview question to ask is if everyone has gone through the same hiring process and if not ask what the previous process was and why things have changed.
I don't think any of those things will increase team productivity at all.
Mentoring and training and encouraging learning new things and constantly giving new challenges and expanding responsibility and tying company financial success directly to employee paychecks, are the kind of investments that can improve a team's performance.
Is it really that people aren't motivated or that they aren't motivated by the incentives you're offering?
> A surprise pay raise, catering lunches, taking them to an offsite, etc whatever motivational strategies you have may provide some short term result
I imagine what might be more effective is actually talking to them and figuring out what they actually care about.
Hiring people is very expensive. Finding and qualifying leads, creating job description, interviewing, on boarding, orientation, training, introduction to and getting familiar with the existing code base. Firing requires distributing that team member's work to other team members.
This can take months and a big investment from existing team members, to go from wanting to hire someone to a fully productive team member. And that's assuming your hiring process never accidentally hires someone even more incompetent than your "poor" team members.
Compare that to investing in improving existing team members to help them improve. Is that really more expensive than hiring and firing?
Also accidentally hiring the wrong people can be devastating. It's better to identify and get rid of bad hires right away.
"The average great software developer will apply for, total, maybe, four jobs in their entire career."
You say that like it's some sort of epiphany, but do you honestly believe that there's anybody who wasn't trying to hire better people in the first place? If the problem is the quality of the people you've been hiring, maybe your boss should consider replacing you with somebody who's better at finding better people to hire, eh?
So my current employer went from a low skill high dedication worker to no worker because I invested in him. I'm going to continue help/mentoring people because I find it fulfilling, but if I'm honest it's bad for my employer. They currently pay good salaries in line with the market. There's no way they can justify giving a person who's been with the company for 10 years 4x salary growth in 2 years. How could they even reasonably measure the market value of his skills changing so much so quickly?
If this dev's trajectory is true, it's also something worth discussing with your management. There might be candidates for this kind of trajectory in your current dev pool, and any feelings contrariwise are to the detriment of your company.
Too often there's an emotional component to how employees are viewed. The employee today will be the same employee forever, if seen through the lens of a limited manager.
I'll say this: if you can do this 2-3 times, you should find your way to an org that can support and appreciate the lift you're able to provide. You shouldn't be holding yourself down to the poor standard of your current employer.
If you think about it from the perspective of the employee, you are working for an employer who is giving you sizable raises, continual career growth, long term benefits, and a team you enjoy working with. You can perhaps maximize your pay by going through the whole interview cycle and ending up with $10-20k bump, but is that worth it? There is no certainty that the next employer gives such large raises or invests in employees.
As for your case - $80k to $300k jump - that is exceptional. It is not going to be the norm. Not if continual investment happens. Based on your example (10 yrs experience), if they kept investing in that employee, they would have surpassed $100k long ago. If they hit the salary ceiling after 4-5 years, then conversations would shift. Maybe more days working from home, etc.
In my opinion, if you see such jumps happening regularly, then it is 100% on the employer for creating an environment that breeds large wage-skill gaps.
I'm curious what the data point is that leads you to think your company's paying a good salary if the hard-working, motivated, skilled people can easily find work that pays quadruple, because that sounds like the market is saying otherwise (unless the missing variable is location, like moving from Nebraska to Silicon Valley or something).
If he was doing the same job I could see why he would get the same pay. If he had taken on greater responsibilities or become more productive though, why not?
> There's no way they can justify giving a person who's been with the company for 10 years 4x salary growth in 2 years.
One of these must be false. According to "the market", this employee is now worth 4x previous salary, so then how can your company be paying "in line with the market?"
Let the market do it. When someone comes with an offer, give a counteroffer and convince them to stay.
There's no way they can justify giving a person who's been with the company for 10 years 4x salary growth in 2 years.
Why not? Markets are better at pricing things than bureaucracies.
That really depends on what you want the code review to achieve.
Catch typos? Yes, a new engineer can do that.
Check if code fits into the existing architecture, adheres to the invariants of the code base, uses base libraries idiomatically etc? I don't think a new engineer can contribute that from the start.
> Creating metrics through issue trackers and time sheets
Which metrics? It's far too easy to create metrics that are easy to measure, rather than metrics which actually increase the business when optimized for (and developers will optimize for / game a metric when it's used to assess their performance).
You learn by doing, and also by example. So they can do the review themselves, but also see what other people pick up on. It sets expectations, and also shows how other people think.
I always find it interesting when I do the same review as one of my co-workers. We might pick up on the same things, but often they are completely different. It shows the value of having multiple perspectives on a team. Not just for review, but for design, implementation, validation, debugging, and the rest. It widens people's perspectives, serves as a means for ongoing improvement and learning, and makes the team better as a whole. We all have different backgrounds and expertise, and having people share and pass on some of their knowledge and skills is highly valuable.
The reviewer is supposed to determine what goes in or not, so how can this be an opportunity to learn from what goes in? They're the one who's supposed to determine that! Is your idea of a reviewer someone who just spectates all code going through? It's like saying people who don't know a subject should grade work from students taking a class in that subject, as it's a great opportunity to learn from what gets turned in.
This is not a given, and in fact, I would not recommend it.
For small code reviews with only one reviewer, the reviewer gives feedback and they have a conversation about it, but with only one reviewer the developer should have the final say. The problem with giving the final say to a single reviewer is you'll get plenty of pointless code changes to suit the reviewer's individual preferences. Don't waste time on what is a perfectly reasonable difference in opinion.
If you have more than one reviewer, but still a small code review, the final say should be with the reviewer(s) - if their is consensus (you could also go with voting, with its pros and cons). Having multiple reviewers prevents the likelihood of individual preferences dominating.
If it's a significant code review, the final say should be with an experienced moderator.
You specify multiple reviewers. That's it.
It's a completely different skillset, and you're probably only measuring how eager people are to have breadth of knowledge or how familiar they are with your specific system.
Similarly, the explanation for "Debugging and troubleshooting complicated issues" is a bit odd. Why is knowing where the log files are, and knowing how to do complicated test setups for your specific environment a performance measure. Again, that's not really measuring skill but familiarity.
The other two measures are just "Do they complete tasks" and "Do they follow code review guidelines", neither of which are very good measures beyond pass/fail.
The conclusion is: > Once a set of skill areas for a role is landed and agreed upon you will want to make sure your team knows in advance what they are being measured on
So, to do well in your business, I need to pick easy tasks, snipe code reviews, make pointless CI tasks and spend all my time learning the build/test processes not actually developing the product? :)
Someone else then gets to do hard tasks and gets bad reviews because they're slower at this and the numbers do not tick up.
But if you measure only ancillary things, that's a pretty bad measure.
I don't expect every one of my team members to understand the entire build system, testing setup, logging system. That's a waste of their time. They should know some, and perhaps one of the areas in depth.
In my experience, this skillet being measured is the critical indicator for high-quality development talent at a healthy shop. You might need one deep-magic wizard for a particular area. You need breadth everywhere. You need to kill "that's not my job" stone dead when you see it.
Writing code is easy. And it's usually an additive process, even when the code is subtractive. All the other stuff, the being-a-person-in-a-community stuff (because that's really what it is), is multiplicative. So I pretty unashamedly hire for curiosity and for breadth. When it's necessary, I contract out hard expertise until curious people can develop it internally.
Absolutely! That is the "dev" in dev ops.
I think there are few things that increases the productivity of a dev team more than improved processes.
> Why is knowing where the log files are, and knowing how to do complicated test setups for your specific environment a performance measure.
Because if you don't know those things you are much less helpful to your product team.
As a technical lead, I feel that I'm able to judge the ability of individual team members, but I'm having a hard time objectively judging productivity. Simple count of PRs doesn't really tell the whole story, and some tasks look simple in hindsight, when in reality it took a lot of effort to find a good solution. There are also a lot of other complications I'm not going to dive into, but the end result is that it's hard to have an objective productivity evaluation based on the engineer's output only.
I'd be interested to know how other people evaluate individual productivity?
One technique I've found that helps a little is having everyone individually estimate tasks before hand. Over time you can notice who is completing tasks faster or slower than the average. Of course this is just one data point that needs to be weighed against a variety of others.
I would like to find a feasible approach to productivity evaluation using the output only (PRs, reviews, basically all the data points the article mentions), because I feel that’s the only way of creating an environment, where team members can proactively take a bit more time when it benefits the end result, without fearing any negative repercussions.
And you would notice if someone's estimates are always higher for the work they are assigned compared to others.
You, a human being, would never actually confuse the engineers with 0 PRs because they spent the last month playing online poker with the engineers with 0 PRs because you entrusted them with developing an automated deployment system for a legacy application and starting a mentorship program.
Many metrics will spot the outliers. But what is the business value of knowing when an engineer rated 82.13 vs 84.51?
The problem is that if you don't have an objective approach to evaluating productivity, all sorts of biases come in, e.g. if I see someone coming to the office at 11am and leaving at 4pm, I would subjectively rate their productivity lower, even though objectively the results might be the same as for someone who comes in early, and leaves late.
Metrics are a distraction unless in the service of answering that question.
This article contains dozens of grammatical errors (starting right off with the title). Whenever I read something like this, I'm so distracted by the errors that I can't even focus on what the author is trying to say. The English language is dying.
This article reads like a series of bad advice from 1990s.
- No, engineers shouldn't be writing code that "adheres" to specifications. They should study, understand, question and contribute to problem statement and approach at all times (also called "requirements" in pre-2000s).
- Manager shouldn't be at center of evalauting performance but rather establishing process, standards and collecting feedback and metrics.
- Bonuses are inherently evil and would always motivate individuals to exploit short term gains at the expense of long term sustainibility. Any performance evaluation strategy must keep this issue front and center at all times.
- Large part of performance feedback shouldn't come from managers but peers
- Performance reviews should never be entirely metrics-driven. No finite set of metrics tell the full story and all metrics are susceptible at gaming.
- Don't treat new comers as incapable of fixing bugs or do X but not Y. Don't create class system of seniors vs juniors. Titles cause more troubles then they are worth.
Tie their compensation to the product's financial success.
This means, to the greatest extent possible, everything relevant to the product's success must be owned by the team and become their responsibility. Maybe there are some cross cutting concerns that should be the responsibility of a group separate from any product team, but those should be rare and require a strong justification.
This will have an amazing effect of clarifying prioritization of what to work on and figuring out how to deliver it as quickly and reliably as possible. Suddenly the whole team will be in the loop about what features are most important to the customers. Suddenly the things blocking new features demanded by the customers from shipping will be cleared away.
I think for any other metric you can devise, either intentionally or unintentionally, employee behavior will be optimized for satisfying the metric, and not customer satisfaction with the product.
This is an atrocious idea in anything much larger than a start-up. It leaves the engineering team beholden to bad decisions made in other facets of the company (sales, marketing, upper management). It's exceedingly frustrating to write a solid, stable, performant program only to have marketing or sales push it as something that it is not and nosedive the company due to customers calling bullshit.
Start-ups are somewhat immune to this because of the lower barrier of communication between departments, as many people will be wearing multiple hats due to there being more things to do than people to do them.
This is very visible when executives begin choking off benefits across the board. Nothing kills morale quite like losing your bonus because another department isn't doing its job properly. I've seen more than one mass exodus from a company as a result of this.
I mean, you're screwed anyways if those people can't do their jobs. No sales means no revenue means engineers taking a pay cut or getting laid off.
So the sales and marketing and product management responsible for your project, need to be on the same product team as the engineers, and have their compensation tied to the product's success.
If the engineers see themselves as the natural enemies of sales and marketing, the product is probably already doomed.
Having team members focused on getting themselves ranked higher than their teammates, means incentivizing them to not help each other or do something they might not get credit for, even if it's in the best interest of the product or the team.
Could add in factors for customer retention and customer satisfaction, for example.
Everyone on the team should share responsibility for figuring out what will make the customer's happy with the product and how to deliver on those things.
The point is that measuring performance off of single metric is incredibly tricky. In the field of reinforcement learning this is known as reward function engineering and its one of the hardest thing to get right for even well specified problems. Employee actions would need to have short term as well as long term focus (exploitation vs exploration). Most actions have consequences that won't reveal itself completely until sometime in future. The current "state of the art" technique is to use stock vest schedule with hope that employee would care to grow his/her future stock vest.
- coolness (cooperation, proactiveness, professionalism)
- carefulness
- integrity (ethics)
- morale
- cadence (speed) of work
- skills competencies (matrix)
- grit (badassery)
- estimated time to completion multiplier
The best strategy after looking at various strategies under simulation is to focus on developing the people you already have since fixing your quality through hiring is very difficult and expensive.
Managers will argue that they have so much responsibilities that they cannot code together with teh team, but I again would say that the problem may be reduced with reduction or higher management. Bussiness part shall talk with engineers on feasibility of their vision and ideas without proxies who are so in the middle that they neighter understand bussiness nor technology.