Engineering management lessons (2014)
defmacro.org
defmacro.org
* Protect your engineers' attention.
As a manager you are the primary firewall between the world of distraction and the world of getting shit done. Most places these days have institutionalized some amount of distraction -- the various recurring "ceremonies" -- but anything beyond that, it's essential that the manager can protect the engineers from being distracted when they're working on hard problems.
(I've been pretty lucky in this regard throughout my career, but now that part of my job is to live among the distractions it's really been driven home.)
Also:
> You’re the one who makes hiring and firing decisions. Everything that happens on your team is your responsibility.
This is, to put it gently, not quite true in larger organizations.
Can you elaborate on "attention", and what does "protecting" it look like? (I've seen this reasoning used to keep developers out of requirements gathering, for example.)
That can be fair reasoning if you already have difficult work to do and your customers are noisy, disorganized, or inexperienced with developing requirements (think NYC window-shopping for features on someone else's credit card).
On a serious note, I have seen more projects ending into dumpster because we listened to customer too well and didn't actually infer the meaning which lies between the lines.
Most managers I have encountered are just messenger pigeons. Sometimes I wonder how system still works!
What you want to avoid is random people (and that includes higher ups) pinging the engineers directly. That can be incredibly distracting and can result in large decreases in productivity, not to mention higher likelihood of bugs and other problems as the engineer is kicked out of their "zone" and loses their concentration.
http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-...
I've worked in larger organizations where communications got funneled through so many managers that any problem, once it was a cross-team or cross-department issue, got lost in a giant game of telephone.
A pretty significant issue that could have been solved by a single configuration property change took nearly a month to accomplish just because the problem got lost through a miss in communications in one link of the chain.
The problem is when requests for larger ideas and tasks do not go through your PM/PO (they should be protecting their team from "malicious" requests so they can meet their milestones on time).
For example, someone from product or marketing shouldn't be able to walk up to you randomly and request that you build something for them. (When they do, redirect them to your PM/PO.)
My attention is my ability to concentrate on what I need to do right now -- not what the company needs in a broad strategic sense, not what the stock is doing, not what might be on the horizon in a few weeks. All of those things are important to me but my attention can only be on one at a time. And I think the reality of serious software engineering is that sometimes you need to concentrate on the code and not on the other things.
Protecting my attention means knowing when I need to be in the "zone" (asking if you aren't sure); giving me plenty of warning if I'm going to be involved in things like requirements meetings; totally insulating me from any corporate politics except where I've asked to be informed or it directly affects me; and escalating appropriately (HELO phone?) if I'm too busy to answer e-mails/chats.
I absolutely think you want engineers in requirements meetings, but I also think that if you have an engineer who should be in the requirements meeting and she's super busy implementing that autonomous social-graph API then hey, move the meeting.
I have a rule that I'm the gatekeeper to my group's time. My people know that if someone tries to contact them directly (unless as part of an ongoing project/conversation), they're to direct that person to me, where I can set expectations appropriately. It doesn't happen much anymore.
I see part of my role as a liaison between my group and the rest of the company.
My people don't fight fires, there's a team for that. I'm sorry that you don't feel like your issue gets the attention it deserves over there. Those guys are swamped. Pinging my team directly is not the appropriate resolution.
Usually, what happens is someone from some other group (sales, support, services) will come to me to ask a question. If I don't directly know the answer, I'll throw the question in a Slack channel. My ask of my team is that they check that before they head out to lunch, head home, etc., because it's usually a 5-minute investigation.
If your engineers and builders are focused on the crisis of the moment, they aren't building.
Fundamentally, people don't do multiple different types of tasks well at the same time. Let the builders build, and the operators operate.
We had a "visionary VP" who'd visit the devs personally. I placed my desk at the entrance to our area, so that I could catch and block him (and others).
We had a basket of shiny toys (eg books about XP) to keep this VP amused, satisfied that we were sufficiently forward thinking enough.
For my part, I've always moved my devs closer to the customers, bypassing the internal hierarchy. More signal, less noise.
This might be the single best example of "managing upward" that anyone has ever written on Hacker News.
I've seen so many companies regularly invite engineers into meetings or have multiple disruptions throughout the day.
While I agree with the general point here, I'm not convinced priorities are necessarily the right was around. Talking to customers/users usually seems to end up paying off in the long run, even (perhaps especially) if it's a bit rambling. On the other hand, it's the regular ceremonial stuff that leaves me agitated and unable to focus afterwards.
More often than not we end up solving the issue via email.
Most of our "engineering disasters" were related to lowering our bar for hiring. Some "engineering disasters", though, i should note were related to inexperienced product development - so if you're a first-time product leader, get an experienced mentor.
Let me guess, Amazon?
I agree.
It's easier said than done, though. Many engineering managers aren't really involved in the code in any real way. I can't imagine running a restaurant and not inspecting the kitchen, seeing hosts greet patrons, seeing how well tables stay cleared of dirty dishes, and so on.
I find that a lot of "hire the best" ends up boiling down to knowing which engineers are popular with "technical" folks and haven't obviously messed up (at least to someone who hasn't really looked at code or deployments). I'm not sure how to keep this from coming down, again, to politics and popularity.
Everywhere I've worked in the last 15 years has aimed to hire the best, none have succeeded.
A "hiring the best" motivation can easily become anti-diverse, because it plays on people's biases about what constitutes "the best", it can also foster a culture of arrogance
Do we ever think about hiring good enough for the tasks at hand?
If you have a great attitude and you're able to learn then you will be amazing on my team.
It's impossible to say "hiring only the best". Because where are these "only the best" people? I've been smashing keyboards for 35 years, does that make me "only the best". Nope.
What makes an amazing team is the ability to learn, adapt and solve real world problems at scale together.
Not "only the best".
At any organization there will be people of various skill levels. Does hiring the best mean you never hire juniors? It is the best at any given experience level? It might even be hard to compare some people. E.g. what if you have two people, both excellent, but excellent at slightly different things?
Rather than saying "hire only the best" I would perhaps instead suggest that you hire people who fit your culture and compliment your existing teams. A team that works well together, doesn't fight, doesn't get distracted, and is happy, is going to be efficient.
Or, another way to look at it is to consider the "Secretary Problem" [1]. When do you decide to hire someone considering that you can get a stream n people applying over time? You don't want to wait until the last person since n can be large and you need people now not in a year. The solution is to not hire the first n/e people, then pick the first person who you deem to be "the best" compared to the first 1/e people.
Hiring is complicated!
It takes a good deal of ego to not cave to the pressure, and a growing ego feeds on accomplishments (ok, mostly ;-).
You want to make it to the superbowl? Well that is easy just have the right team.
Great advice.
Probably not.
OP means you should be raising the bar for hiring at your org.
So what's the bar? How often, in feedback meetings is the criticism technical versus non-technical?
I know I never get specific technical evaluations from managers. Ever. No exaggeration.
I'm not talking about the mythical 10x engineers or anything like that. It's your bar for what skills fit into your company. If you have no bar...you've got much bigger problems than management...
That said, if you lower the bar a bit in hiring, do not lower the bar in expectations. See if that person who you weren't that confident in can catch up fast. Some candidates work very hard and make it. Those who aren't are apparently fairly quickly...you have to let them go.
Incidentally, there is no real "best" when it comes to engineers -- I've found there are two ways to think about the engineers you need: specific skillsets and brilliance. Someones you need someone who really knows a particular technology or area and can do that very well even if that person isn't generally super-capable. Such a person is often much more efficient than any new-comer will be even after several months of training. Then there's certain work that requires learning or general brilliance (which is probably learning fast if I had to define it...).
I disagree. Most problems don't require the best, and hiring the best to do things that might be a little bit simple for them is a great way to promote frustration and turnover.
Every organization I've worked for has wanted to and said they hire the best. They didn't actually hire the best, and that's a good thing, because the work we had simply didn't require that horesepower. We needed very good, so I'd hire very good, and it's generally worked quite well.
> It's at least as important to test for ego than for technical skills.
I agree with this. I won't hire assholes. I've inherited a couple when rotating and taking over new teams, and not once has their productivity, subject matter knowledge or any other technical contribution resulted in a net positive when factoring in the attitude issues and other soft impacts to the team. Don't get me wrong, confidence is a quality I like, but to draw from my username, there's a fine line between stupid and clever.
> so if you're a first-time product leader, get an experienced mentor
This is good advice to any manager at any level. The day you're promoted, you're new all over again, make sure someone experienced is teaching you the ropes. And when you're that experienced person, make sure you're teaching someone junior the ropes. This is why I resent climbers/gunners in the organization - they ignore the latter part.
* production vs process - you can spend resources on improving process(usually tooling) to improve production but this isn't always an easy knob to tune and can have diminishing returns. "Automate all the things" is how I know you don't know what you're talking about.
* DOTs - damage over time can be brutal, question every meeting and email that hits your team. Similarly- Engineers need large blocks of time to find flow and be effective(aka void rays). That 15 minute 2pm weekly check-in may seem trivial but you're pretty much nuking an afternoon's worth of productivity.
* macro vs micro - you generally need to spend time on both. Even Sr. engineers can benefit from occasional guidance because you should know more than them(see below). (knowledge != intelligence)
* bottlenecks/'stage III' production - you can have too many resources on something.
* Some resources are much better at certain tasks than others.
* situational awareness - know the executive roadmap, your roadmap, your internal customer's roadmaps, know 'the theory of how engineers are working in your org' and the reality. Know every project in your code repository. Know about most other possibly relevant open source technologies. Mostly this is just a lot of listening and reading. Be Omniscient
* 'strats/build-orders' - similar to the above, know how the industry/competitors solve the same problems your org has (but don't necessarily seek to copy them)
That said, treating people like RTS units is also a good way be disliked and ultimately be ineffective so you have to do the whole empathetic human thing.
In a good software team, a single developer's "big mistake" gets fixed by the whole team. I guarantee nobody on that entire team will make that same mistake; nor will they hold anything against the offending developer unless there's a pattern.
But in short, empowerment is the most effective leadership style in this environment. It's not appropriate for all employees (specifically newer folks who need a bit more hand-holding at first) but for professional workers like engineers, it's really the only way to go -- especially if you have senior technical employees who have decades more experience than their manager.
Just not "Personally fix bugs and ship features."
Then about the 5% - this is high prio, high visibility fixes where you bring in your extraordinary expertise in some domain. Believe me, that's enough to stay respected. Your time is much better spent at sparring systems architecture & high level design, hiring and building well-oiled teams and discussing the business needs with the rest of the company.
Isn't as fun, Doesn't feel as good, but it helps most the long run. The alternative is a high performance coding monkey that gets a tiny area of the company right - and the rest of IT goes wild and rewrites core APIs in the latest three hype language because "we needed to get rid of technical debt".
The 44 lessons of the OP are distilled gold from years of experience, I rehearse them every year and can recommend them to any aspiring tech lead.
How hard is it to answer that question?
I just ran into this response yesterday, and I ended up going over his head (unfortunately) and asked our DBAs for a recommendation. DBA solution worked faster than either of the previous two solutions. I still need to apologise for that one.
Curious. Running an idea by people to get their thoughts, because it's currently leaving you short on warm fuzzies... is something to apologize for?
The initial solution was low on warm fuzzies for him. Your second solution was low on warm fuzzies for you. So you explored further. Where's the problem?
The army figured it out. The Officers are in charge, but the sergeants get shit done. In engineering, your leads/supervisors are the sergeants.
Officer quality varies between complete shitbird and Julius Ceasar. Either way, the army marches.
Their main weapon is pestering you.
If you come from a strong technical background when first promoted (Being the go-to expert on the team or similar) then you can probably keep the respect of the team for a while but eventually others will rise to fill the gap (After all there is 95% of a rockstar programmer's time missing in the team). At that point your comments on code reviews will mostly be seen as micro management.
I would say that there are two ways to go about it, either fully trust your developers and let them develop and review their own code or stay as an active member of the team, doing as much coding as you can fit into your schedule. Which route you pick of course depends on the workload of managerial duties.
Reading lots of code also gives a broader perspective than writing lots of code, which requires a sort of tunnel vision. Both the deep and the broad perspective are useful on a team.
The amount a developer (not manager) should code varies from domain to domain. In my work, 5% may be on the low side. But I'd argue in most domains, if you're spending more than 20-30% of your time coding, you're overdoing it and creating too much technical debt.
I'm not a manager, but if I were, I'd promote the folks who find (good) reasons not to code than to code. Some problems need software to solve a problem. But writing software should be a last resort.
Of course, if you're writing throwaway code on a code base that won't last more than a year or two, code away. Technical debt is not a big concern.
I honestly don't care if they code or not.
This does not apply if you are a <10 people company though.
A possibly better approach is for managers to have a few side proejcts, scouting out potential risk reducing things, at a higher temperature, away from the main project. Great if one of them lands - no added problems if not. Also, if they land in the project, they are no longer the responsibility of the manager.
It was never an issue to us and they were always respected by the team, except for the few ones that never should have been managers to start with.
Any manager that wants respect from the team needs to support them, specially against stupid decisions from upper layers. Have good social skills and specially important, how to raise morale in darker times.
Knowing to code is irrelevant.
Mostly enterprise size companies.
One big thing I've learned is the manager should never take on big pieces of development. The other demands on your time mean that every task will take longer and you'll end up being a bottleneck.
If the manager is required to complete the feature their "free" time is't the same as an engineer - read meetings/discussions. I manage two teams and when I first became a manager my instinct was to take tickets along with the team. Yet my other responsibilities came in play and made it harder for me to finish tickets I assigned to myself and others on the team waited.
I still code review and partake in high level design and some scripts here or there etc. but not mainline code that another engineer will depend on to be completed by xyz date.
The goal of a manager is to ensure the work pipeline and project trajectory are on path with the company needs while still taking time to learn new tech and skill-sets of the team. Of course most importantly protecting their time from distractions.
To move from manager upwards, building a great team that trusts each other and follows and ensures the team standards is crucial. If I am required for every decision or code review, then I become the bottleneck. Empowering other people and trusting their judgement was one of the harder leanings growing from manager upward.
When I was a vp eng. what helped me the most was having ONE feature that I owned. I was forced to use the build system, go through qa, fight with design. It was never something massive, or hindered other groups, but if someone on the team bitched: "this framework sucks because.." -- I was able to jump in.
> Don't supervise the quality and volume of people’s work.
And I agree with the argument following that sentence. On the other hand, 2 points above that it says:
> Do enforce behavioral and performance standards. Fire bullies and underperformers.
So how else do I recognize underperformers without taking note of quantity and quality of people's output? Surely I can't just take someone else's word for it (and by the point a peer complains about this it's probably too late already anyway?).
What if you have an employee who doesn't know how to do software estimation? Do you know of ways to teach people how to do estimates or resources/habits they can follow to learn how to estimate how long tasks will take?
I ask this as someone who has only received the advice "go with your gut and triple it." There is a book by Steve McConnell on estimating larger projects but nothing out there on estimating things within a week.
If the problem is in fact known, your estimate will be the time to pull in a library from GitHub. If it's not to pull in a library, you're into unknowns.
Asking to estimate may work in aggregate if your 'gut' is good at assessing the amount of unknown your devs' face (e.g., 2x as much unknown work - multiply estimate by 3), but not in the specific. The specific amount of unknowns in a given task are unknown.
To translate a requirement into code will take exactly as long as it takes to discover and bridge the unknowns. You can't do anything about that, other than (a) change the requirement, or (b) change how much is known by the developer.
My rule of thumb is that if you cannot estimate your work to some degree of confidence up front then you haven't spent enough due diligence time and the design before writing code. If your task is too complex to estimate then estimate a time boxed proof-of-concept and then turn around with a solid estimate at the end.
Its my personal belief that any programmer who expects a check from an employer or a client for unbounded unscoped scope work usually isnt unreasonable. Just like any other profession.
But, you can look at a piece of code and go "no that doesn't make sense. Lets try this instead" and both reason about its quality or run tests on it. You can have nuanced discussions of what makes a piece of code more clear or a tutorial code example more easy to interpret.
It seems like no such nuance is available.
I share your opinion that it is unreasonable for me to expect to be paid without giving an estimate, but I hate hate hate lying to people and so I either avoid giving an estimate or I give one and try desperately to convey "this number is a near-random guess." I have on occasion failed to impress upon someone the uselessness of an estimate of mine and the deception makes me sick.
So I don't know what to do. I have occasionally considered giving up programming, but that seems drastic. I'm currently in a company that doesn't do estimates that much or put too much stock in them and has quite good habits for clarifying scope. But, I feel really unprofessional that I can't estimate.
For 10 years I was a small gig consultant 5 of which I worked solo. Fixed price contracts. Clients with tight budget. Projects with high technical variety.
People like us our product is our time. Getting things done on time/budget (or not) sometimes yield a significant impact on our reputation. Thats pressure especially when youre working with so many variables!
Everybody in this line of work goes through this on some level. Based in my downvotes above i assume many folks dont like the idea of trying to estimate to high accuracy. Thats their take. Everyone has their own opinion. Heres mine:
Itemize out the work into logical work items. For each item fix the estimate and the level of confidence on your estimate. If the level of confidence is below say 80% then state the unknowns and add estimate an optimistic/pessimistic range for the item instead of a single number.
After going through this exercise bring this to the client. In all my years doing this ive never had a client who didnt appreciate a thoughtful breakdown of the work and it became easy for all parties to appreciate what areas of the project had the greatest unknowns and why. Also the client can either accept the estimate based on the pessimistic high end of the range or we find a cheaper alternative or i scope a proof of concept. Either way the choices are clear to everyone even if the estimates arent entirely clear.
Sorry for the long-winded explanation and thank you for reading. I would summarize by saying an estimation is just that. it is not accurate but the unknowns are clear and the work is carefully considered before beginning in earnest.
Best of luck to you and don't give up
I'm highly sceptical of managers that are focused on measuring metrics.
If the engineer can't convey why it took a week to find that one-line, they should work on their communication skills. A good manager can help set expectations for an engineer who has underdeveloped soft skills.
I just spent a week and a half tracking down a bizarre issue in a 3rd party library and filed a fix for it. It was simultaneously frustrating and fascinating, but I never once was worried that the team didn't support my effort because I kept the team in the loop during daily standups. I've also been lucky to have excellent managers who believe in letting the engineers work.
On the other hand, if somebody is constantly disappearing for a week+ with a low complexity ticket, gives cagey standup updates, and then submits a PR that looks like it was thrown together in an hour.. That's a red flag that needs attention.
Spot on, a good leader does not make all the decisions. It is counter-intuitive to popular culture, but this is stop on.
I would go on to say minimising your decisions, also helps to spend more time on a key decision when it is really necessary.
The deep emotional drive is also what causes startups to succeed where big companies fail, despite their massive resources.
1) Provide Air Cover
2) Provide resources
3) Provide direction
4) Get the F* out of the way
Don't over think it. Empower your teams, trust them. They'll make mistakes just like you do. It's not how bad you F* Up it's how well you recover.
Originality in mistakes is the best I can ask for. Let's learn, progress and do better together.
I've been in tech for 35 years, if you think you've seen it all, you haven't and neither have I, so let's figure it out together and get some S*t done.
Every engineering team has problems they can't deal with on their own, either because it's out of scope, it's high-level organization, or whatever. A GOOD manager spends a significant chunk of their week ACTIVELY fixing problems. Not getting-the-fuck-out-of-the-way!! They should help clear the way. They should help fill gaps in the team, plan, or execution. Talk to other teams and see if everyone's on the same page. Make sure everyone agrees on priorities. Make sure your team's work integrates with what everyone else is doing. Be aware of everything going on around you (esp with other orgs, etc) so you know how it all fits together and your team doesn't waste time.
Don't just listen and nod during 1:1s and then give some off-the-cuff suggestions. Don't just "get out of the way". And you don't need nearly that much Air Cover when you're team is aligned with the rest of the company, so make sure THAT happens instead of trying to "protect" the folks that happen to report to you.
If a great team's manager can't point to what they ACTIVELY DID to get the team to success, they're disposable.
I've had managers who couldn't demo the apps that their team worked on.
Which one is better to work for as an engineer? The technically savvy one. Which one has higher stature and seems to be more highly regarded by his superiors? The non technical one.
In other words, that "lazy shitty manager" is probably playing the game better.
There are many things that can often use improving, especially at smaller companies. Process, architecture, deployment system, reduced meetings of team, optimizing the SLM software...it's easy to get overwhelmed, especially if you have a large team, or in my case, doing the job of a tech lead and engineering manager. It is a lot to swallow. Prioritizing the most important points on what is being tackled and communicating why is very important.
That said, I only counted 6 items out of 44 that's specifically "engineering" related. (5, 7, 10, 12, 14, 15)
The other 38 bullet points are universal lessons that also apply to non-engineering managers such as a director of a creative team in an ad agency, or project supervisor of a construction crew, or a showrunner in charge of scriptwriters for TV episodes, etc. Empower your team and make decisions when necessary, etc.
The art of managing relies on the capability to get work done by all kinds of engineers. The talent is rarely universally allocated. Some very 'bright' ones will never properly finish the work. Some 'slow' engineers may have remarkable attention to details.
I saw this phrase several times in this article and the linked article on "non-violent communication," but I don't think it's a particularly productive phrase.
Consider the possibility that no one can "make you" feel anything, and that you are ultimately responsible for your own emotional state. For example, you can't command me to feel happy or sad. My emotional response is something I'm a party to, in combination with my own unresolved fears and insecurities.
Instead, I suggest phrasing it in the present tense: "When you do X, I feel Y." That is observational and avoids any accusation or blame, so you can focus on the core issue at hand.
Personally I'd like to see the top management get fired.