Senior Engineers Build Consensus (2019)
hyperbo.la
hyperbo.la
Wadge’s Law (of Meetings).
Before every formal meeting there’s a smaller, more exclusive, less formal meeting where all the important decisions are made.
This is based on decades of experience in academia and friends’ experience in industry and government. Sometimes there’s an even smaller, more exclusive, less formal pre pre meeting where all the decisions of the pre meeting are made. Maybe even a pre pre pre meeting … until you reach some guy deciding everything in the shower.
Likewise, if a meeting is where you adjudicate something, you need consensus to focus on the key issue, whatever that is. Otherwise, you’re likely to head into some rabbit hole that results in no decision.
The problem being solved is that most ideas are not good, so any single person with an idea looks to vet it among a trusted group of advisors/peers. If this group is too large, it's hard to deal with the noise, too small and it may kill or ok an idea when it shouldn't be.
After refinement with the smaller group, an idea now has enough substance to bring to the larger group and hopefully not waste their time.
A simple example this process helps avoid would be pulling together the full group, presenting an idea, and then legal killing it with their first comment. Everyones time was just wasted since the idea as presented had legal issues and needed more refinement.
You can totally run things in authoritarian manner, gathering consent in instrumental way and use outside sources (like sales) for validation. But downsides/side effects/intentional features not said out loud are well known and researched to the point where one has to intentionally choose to remain ignorant of them to sustain a different narrative.
The other key aspect is that it's just far more efficient. My last few jobs have espoused being highly collaborative, but in practice what that means is that everyone winds up in the proverbial room. Chaos ensues, nothing ever gets decided.
So at the meeting, only hitches to the (expected) plan are expected. Not building a plan from scratch. Its more like a standup than a bull session.
https://files.manager-tools.com/files/private/podcast/mp3/ma...
The manager’s new tactic seems to me both more effective and kinder, because it takes into account the engineer’s need to process a change outside of a public setting.
In the month leading up to that, interactions at work were also giving me anxiety bordering on panic attacks, like a visceral fight-or-flight response. That turned out to be my atrioventricular nerves progressively failing, such that my heart ventricles were beating slower and asynchronously from my natural pulse rate. I got down to 21 beats per minute at one point, and everyone at the hospital was amazed I was walking around and smiling rather than on the ground unconsciouss.
I'm pretty much back to 100% now, and as an embedded software engineer, I am horrified to be able to say that I'm bluetooth enabled.
Also: this is wildly off topic, but did your 8 hours of sanding off cracked anti-reflective coating work? What equipment did you need? I had a similar issue and it never occurred to me that it might be fixable at home.
I sanded the lenses by hand with 200 grit, then 400 grit, then 800 grit wet sand paper. There was the AR coating, but then under that there was some sort of underlying "toughness" coating that I had to completely sand through to get down to polycarbonate. That's what took the longest, since I didn't really know how far I needed to go with it, and it was fairly resistant to the abrasion. I tried soaking the lenses in isopropyl alcohol to soften the coatings, but I don't think it helped at all vs. just applying elbow grease. Once the surfaces were uniformly smooth but frosty looking, I polished them with Novus plastic polish #3, then #2, then #1. It wasn't perfect like a new lens, but the center was optically clear without any starburst reflections. With all the coatings gone, the polycarbonate quickly picks up little scratches just from cleaning, but it's quick to polish out again.
It was worth it to keep me from going completely insane for the next month it took me to get an eye exam and order new glasses.
Taking corticosteroids briefly then stopping cold turkey like that is pretty rough on your endocrine system. I had to be on the high dose for many weeks to get the desired effect, and my doctor had me slowly taper off over the course of a few months to keep from crashing like that.
I will often just tell people okay that's a big change give me time to go through the five stages of grieving and I think we'll be good
Btw one reason companies ask interviewees about their weaknesses is not because the answer matters but because it shows whether a person thinks about what they are like, or not. Eg if someone ever said what you said "one of my weaknesses is that I freak out in the first 5 mins but I learned to work around that" - a real green flag that you are an excellent person to work with.
As a manager I learned to keep my senior engineer pre-informed the hard way. Personally, when I was an IC, I was totally fine not being kept in the loop because I was impervious to such news or changes. So I just assumed that’s how it’s with everyone. Clearly I was wrong.
That said, it is important to release that pre-information in an informal fashion lest they start acting on it before it’s formally announced. Especially ones that impact the immediate peer teams such as re-org.
Also, I don't really see the point in hiring supposedly intelligent engineers and then cutting them out of the decision making process.
Btw I'm not even a senior engineer, I'm a _junior_.
To be explicit, there is absolutely a place to “shield the team”, but I’m talking about a pattern I’ve observed that many managers will use that as a pretext to avoid keeping people in the loop and essentially get to boost their own career by holding others back. Real shielding is when you politely tell team Y that their proposal is going to require your team to have to own and operate a net new production service, not when you make critical decisions without consulting some of the biggest stakeholders - your own team.
That's clearly not done. What I referred to was decisions that were taken outside the control of the line manager; such as re-org, change in promo process. The line manager will not be part of such decision making process they are mostly taken at director level or above. The line manager though will be kept informed/warned of such changes about a few weeks in advance which is when I would start warning senior engineers in my team.
You will be surprised to learn that line managers have very little influence or say in most of the critical decisions taken. One of their responsibilities is to keep the team pacified i.e., to ensure the "shield" works both ways ;-)
In a biggish company you will come to learn that ICs's opinions doesn't matter; at best they will be heard to be ignored later. So as an IC the best thing for me was to focus my energy and skill to create the value the best way I could i.e., shipping software and helping my immediate neighbours unblock their work whichever way I could. Once my line manager noticed that I was indeed creating value through my primary skill he began seeking my opinion once in a while and I could see that it did have a bearing in his final decision.
There were a few vocal engineers who would regularly vent out their frustration over mailing lists, lunch tables etc., But as far as I could see it wasn't very useful in the sense it wasn't actionable. It takes quite a bit of energy and time to influence an outcome in a meaningful manner. You first need to build up sufficient social capital and then start pushing your opinion either directly or through others (i.e., "influence the influencers" as they say). Obviously it is a useful skill that can be acquired, the question is will you enjoy this in the long run.
I'm also more interested in understanding and being involved than simply being frustrated and venting.
I didn't realize it at the time, but everyone of the same seniority level, but a few years older, already realized not to give a single fuck about anything beyond their immediate sphere of influence. They had already learned that the work truly did not matter, and at a moment's notice they'd be tossed aside as person if given enough reason.
So, don't hope for much if you can't really control it, or spend your time gaining control, as in agency, over decisions that are actionable. Otherwise, burnout is on the horizon, because your expectations and devotion are way higher than you might get out of it, but the stress is maxxed out.
The parent here is perfectly on the money.
I'd also extend this to other areas of life. Probably try not to involve yourself too much with things that you can't influence, or things that you think people should care about but really don't. Not that they aren't inherently good, but you'll grind yourself down trying for nothing specific.
I'm not quite foolish enough to think that I will be able to control how things work no matter whether I'm a junior or senior. Even if my involvement was limited to observing, that would still be a win to me though.
> Probably try not to involve yourself too much with things that you can't influence, or things that you think people should care about but really don't
I appreciate the warning, though I think that I've already cultivated this type of attitude. I have a healthy amount of cynicism towards certain things thanks to reading about other people's experiences.
By the way, I'm curious what sort of companies you have worked in. Non-tech companies, tech companies, FAANG, etc?
Your advice is contradictory: you’re just immediately causing future pain.
It is infeasible for literally everybody to be involved in a decision. So then the question is, how do you communicate a decision and when?
If you communicate "we are thinking about doing X" then this often causes problems with interpretation. People read these as big shifts in direction and experience whiplash or get overly excited and then disappointed when X doesn't happen. If you communicate "X is happening in 3 months" then the people negatively affected by X just get more stressed for more time even though they can take no action.
This can be different for different individuals. If somebody is especially bothered by not knowing things ahead of time you can adjust. But I find that if information isn't actionable then it is often in appropriate to share it way in advance of an actual change.
The engineer in this case had an emotional control issue: they reacted badly to anything new, even if they had no rational reason to do so (they were fine with it later). This could have been tackled by working with the employee to get them to understand that this type of loud public reaction is causing problems for everyone, including the perception of their own skills, and that they need to learn how to take a deep breath when a new change is announced - maybe wait a few minutes to write down what they wanted to say and then wait a few days before hitting the send button. Lots of approaches. Instead everyone else adjusted their behaviour to avoid tackling the underlying problem.
The manager’s new tactic seems to me both more effective and kinder
Maybe I'm just some asshole manager but I never saw it that way. You externalised one person's problem onto the whole team, who now all have to be aware of this special exception. Most obviously it makes it difficult to have brainstorming sessions, or if someone comes up with a new idea half way through a meeting unexpectedly, they can't raise it there and then, they have to wait for the meeting to end, pre-brief this one guy, let him/her get over it, then raise it with the rest of the team.
So whilst it may have been kinder to that one specific person, I'm not sure it was kinder to everyone else, let alone more effective. Especially because once such a culture is embedded, sooner or later half the team has some weird quirk that everyone is expected to work around or ignore.
I've recently joined a BigCo (as a senior+ engineer), and the culture here isn't building consensus (out of authentic building blocks) - it's a toxic "we must be consensus after every meeting."
There's always a "champion" idea (but you can bring a "challenger" idea so people feel heard), there's always a need to "be in alignment" after every 45 min chunk of time, etc.
Consensus is clearly an end in itself, not a means by which we solve problems.
I'll just slither around in it though. Talk the talk so no one gets wise $_$
BigCo isn't fratty at all in my limited experience. Be professional, follow whatever 10 commandments the C-suite prints out & laminates for you, and sell yourself once a year in self-review and you'll win out and barely work if you're remote.
Smaller companies have very little org-chart structure - it's a single small tree. There's no equilibrium in the org chart. So a couple bad actors can easily ruin it all. And they did at every startup I was at.
My play is to collect checks and eventually make art with my computer science experience for the rest of my life.
You too! Let this be a lucrative but minor (ergo efficient) period of our lives :beers:
Use your free time post-covid to build up something worthwhile. A cat is a good start!
Love for its own sake is more than enough. But it takes a suspension of disbelief that modernity may have beat out of you.
And to take it further, even if you’re one of those (many) people who thinks the sky is constantly falling with COVID...there is still no reason to wait until “post-COVID”. If you want to build a skillset start working towards that now, not tomorrow
Not necessarily. You want to convince your team or other teams to adopt a new tool or practice? You need to build consensus.
It's a good skill to have if you want to be able to shape your workplace to your liking. It doesn't necessarily have to involve org chart climbing at all.
"let's have an endless, design by committee approach where the backend dev with no knowledge on UI decides how the UI should look rather than letting the UI designer create a few samples and iterate and everyone else giving feedback like a user would"
or
"we already decided this is going to happen, but for the sake of formality we want you to agree with it so we can feel good about ourselves bullshitting one another into believing everyone has a say, so give us the ok or we'll pester you until you do"
It also stimulates the idea that disagreements are inherently wrong and nothing should happen while a disagreement is in place. Sometimes, it's ok to disagree with someone else and see what happens. Some might call this a form of consensus, though I've had managers get uncomfortable when I didn't vehemently agree with the plan, but was willing to keep my nose out of it and focus on my own things so others could take the risk.
In fact this is really the only way to get anything done regardless of what you call it.
As general commentary - if you're smart and good at identifying problems with things, use your ability to identify problems to avoid those problems, not as an excuse to avoid doing stuff. Being cynical about everything isn't a terrible strategy, as long as you remember that you need to be cynical about your own cynicism, too.
What is the issue?
except it isn't
the principle is "have backbone; disagree and commit"
there's checks and balances in the principle's words themself. every leader i've had who has "disagree and commit" as their guiding light has just been an autocrat. they never mention on the "have backbone" part.
i had a VPE from BigCos come into a LittleBigCo. he was all about "disagree and commit"
really, he had a nose for dissent. he went around managers who disagreed with him to corral their engineers to his side. he put near-founding engineers on bad projects to force them out.
i tried to have backbone with this guy once. i thought one of his tech decisions was bad (and in reality it was politically motivated to force people out.) i wasn't rude or anything. he had a 1-on-1 call with me, and acted like i was being irrational. i stood my ground and made fact-based arguments. he was losing the 1-on-1 debate, so how did he close it?
"look whateveracct..i just have this crystal ball in my stomach, and it's usually right"
disagree and commit? more like stfu i'm the boss
For example, if a leader had an idea X,and Bob talked him into idea Y which worked out, then the leader was still 'right' because listening to Bob was the right thing.
Let’s say you were right: It was a bad technical decision and it was politically motivated. What outcome can you expect from arguing with him on technical grounds? He knows what you’re saying is right, but he already knew that before making the decision. What you need to do is convince him not to act in bad faith. Try to figure out how to do that, and acknowledge if it’s not feasible.
Not to call you out in particular, but I see this as a common mistake by people arriving in a big company. Things are not as you wish, everything is too complicated, motivations are subtle and usually hidden. The trick is to get past frustration that reality does not match your mental model. (People aren’t being honest! How can I even work with people who don’t tell the truth!?) Abandon your mental model, acknowledge reality for what it is, choose goals that are achievable, choose actions that make progress toward your goals.
i knew he was arguing in bad faith. there was no way to win politically - he was making the decision to shape the company in his own image. so the best i could do was visibly disagree, document his various bad actor behaviors, and share my understanding with other engineers who didn't feel comfortable speaking up.
i just thought the guy was a dick so i felt like stirring the pot a bit. myself and plenty of other respected engineers left one-by-one (he didn't even announce i resigned. people were surprised. he was trying to save some face i think.)
the VPE left shortly after. people blamed the engineering turnover on him haha.
overall i feel pretty good about my short tenure there. i negotiated a nice salary bump when i joined and use it to anchor my new BigCo salary.
^ the words of a person that builds things that nobody asked for and then gets mad when people don't use them.
did my fun pejorative phrase offend you?
also - i & millions others just "use" some software service i built years ago at a previous job ;) i have no trouble getting things in production, making and saving money.
What are the consequences of failure to establish consensus?
In a previous position, a failure to commit to certain key decisions led to over a decade of drifting in some aspects because they were unwilling to draw a conclusion and get on with development. For some of these, the cost of actually designing and implementing the solution was far, far less, than the total cost of all the meetings we had about them, not to mention the lost time. I'm not even slightly joking. It's a management failure at the highest level in not considering medium- to long-term issues, by focussing only upon short-term needs. I would also put some blame, in part, upon Agile as practiced by some organisations.
Consensus is important because you have to have the whole team, or whole organisation, on the same page. Even those who don't fully agree with the decision. You have to have everyone commit to following the decision, even the naysayers. That is to say, the organisation as a whole has committed to a certain action. Which is not to say it can't be revisited or re-evaluated down the line, but that right here and right now, we will all follow the plan.
I've personally experienced 4 flavors of work environments: high stress low wisdom politics (big stakes student project); High stress high wisdom politics (SpaceX); low stress high wisdom politics (Google X); and my current project, low stress high wisdom no-politics (Zipline).
Each different environment required me to adapt my own behavior appropriately to work productively with coworkers. Notably, the only environment that consistently lead me to feeling jaded and burnt out in the long term was the low-stress BigCo environment. I see a lot of truth in what you describe about your current experiences, although I never got quite that pessimistic. If you're happy slithering around the politics and are able to find fulfillment in your personal life, good on ya. That reminds me of a coworker at SpaceX who, a month or two before getting let go, admitted to spending 8 years at a defense contractor doing crossword puzzles at his desk all day, every day.
Building consensus around long term architecture is something like 80% of my current job, and over the years while the org chart has gone from 1 layer to something like 5 layers, I've successfully remained at the bottom. The folk at the top of the org chart are completely approachable, and in a lot of ways are there to serve me, solving the problems that would otherwise prevent me from getting my work done.
If you rock up and go, "we're doing this big thing tomorrow and oops sorry we didn't mention it before" of course you're going to encounter more push back.
On the other hand, if you get people on your team before doing something then they will trust you and possibly even become advocates!
From a pure textbook definition of politics, yeah there's politics everywhere, and folk need to learn how to interact with coworkers in a healthy, productive manner. For folk going into engineering, I really like competitive student projects as a way to learn those skills.
Consensus doesn't always exist. Its great to build it and its great to seek it.
Ideally you can document the pros and cons of, say, moving to EBS storage vs instance backed and stakeholders give you comments with their concerns and you document and incorporate these and given enough stakeholders you can say, this is probably good enough for now and you trial using EBS-backed instances, improve your tools, document your learnings, and go from there. This can happen asynchronously, includes anyone interested, and can probably happen in around 1-2 weeks in a healthy culture. You do this so when there is the next EBS-outage you can point to this and say this was a deliberate decision and make an informed evaluation if the argument still holds rather than avoid going in reactionary circles. 1:1s don't document anything and only incorporate the views of people you think you should be consulted which is likely a subset of the people who want to be consulted.
It's not about building consensus together, it's about sneaking your "consensus" into the group by the means of divide-and-conquire. It's very hard to build a solid alternative consensus (or a defense strategy) if all the opposing points have been voiced independently, and whatever one you can think of ends up with "oh, we discussed this with the other party, and it wouldn't work".
Please respect your team and don't use nemawashi. If you are on the other side, learn to recognize it and call it out.
TL;DR: nemawashi considered harmful
But that assumes the person using this process is acting in bad faith to begin with (they're not pursuing the best idea, but rather their idea). If this technique is used in good faith with an open mind, it's one of the most effective ways to deal with large organizations.
It would be nice to have a workflow for group discussions that is robust against the faith differences. Just like we have specific workflow on voting in politics, doing it independently and resisting some of the human crowd instincts.
Half the time I end up getting mind changed, and the final result, while still arbitrary, is better than any of the original plans would have been.
New folk in other parts of the company ask me what I do, and the best way I can describe it these days is meta-engineering. Rather than take long term ownership of any part of the system, I take temporary ownership of the scary parts that need the most attention, refactor them until they're as boring as I can using a suitably large sledgehammer, and then release them back into the wild to (hopefully never) be someone else's problem again. When I'm not doing that, entire weeks go by just reviewing coworkers' PRs, writing architecture documents, and interviewing job candidates. I'm widely known for bluntly (but hopefully respectfully) giving my opinion when weighing in on technical subjects I think I know about, and it seems to be well received. Pretty much all direct and indirect feedback I get is to keep doing that more. Many coworkers even send me their designs and PRs with an explicit request to mercilessly tear it apart.
I've been addicted to watching episodes of "Kitchen Nightmares" this past month, and I just realized that it's sorta like what I do at work, but with less swearing and the undercooked chicken is lack of regression test coverage. Also, the restaurant doors haven't closed up 6 months later yet!
It’s possible to have a 10x multiplier in “individual output” or whatever you want to call it, but 9 times out of 10 the most valuable engineer is the one who can provide detailed, actionable feedback on patches, architectural change proposals etc but is also willing to get their hands dirty when it makes sense. That, IMO, is what I consider a ‘true’ 10x engineer to be
Upon which the 10x moves to inner emigration and gets work done. For which the "team" gets the credit.
Where are those teams that you speak of? At which companies?
If I weren't at Zipline, I'd probably be at Reliable Robotics, but that's more because I know the founders and lots of other employees there, and they have a strong team that's set up for success.
I'm biased towards robotics companies since I'm an embedded software engineer, though.
There is a big difference from asking for forgiveness, which is often required to get things moving, vs. working in an entirely opaque way.
If a company was starting fresh, which would you choose?
See discussion here: https://news.ycombinator.com/item?id=26271117