The Honest Troubleshooting Code of Conduct
rachelbythebay.com
rachelbythebay.com
They often know where the bodies are buried (or that obscure document stored on a documentation system from two generations ago that was deprecated but never removed "just in case" that describes exactly the system you are currently looking at wondering "wtf").
In an idealised world it wouldn't happen but I've never managed to work anywhere close to that.
Saved for future reference. My company has an exploding number of metagamers with an ever shrinking number of "adults in the room". It's stagnating hard.
I like this article a lot; if I were trying to replicate the community I've got now, I'd probably start with this.
Slack is a great medium for realtime chatting but someone jumping into a conversation without context is exactly where it falls down. "Ubuntu is doomed" is exactly the sort of comment someone skimming to catch up will fixate on and jump to a conclusion - even with a code of conduct...
You need a place for conversations to exist that promotes asynchronous communication. A github issue is a nice starting point since someone can create a summary at the top. The summary can be continuously updated. And an intentionally slower/more thought out discussion can follow.
"RFC: Company$ moving away from ubuntu" is exactly the sort of productive GH issue I've seen at past companies.
Of course GH issues regularly devolve - lots of examples dropped into HN over the years where people get angry/inappropriate at some poor open source maintainer, but it is still an improvement over slack.
For example, mandatory-participation announcement channels that don't actually require any action on the part of the receiver. They're used for every corner of the company to self-congratulate and should be emails instead of notifications.
For anyone who can make these decisions in their own comapnies, if a channel's messages:
- do not require immediate attention, and
- do not require an action,
You ever, you know, just talk with your coworkers? And not just sling bugs at them?
All large organizations have varying degrees of dysfunction, some more than others. Sometimes that dysfunction is toxic, but other times it's just a result of people being human with different opinions of how best to do things. Coming in to a new organization and making an announcement that is some version of "you're doing it wrong", unless you're very high up in the org, is rarely successful.
That is not how "Did it work? Kinda" reads to me.
I didn't get that take from the article at all; I've been a part of one of the orgs she's describing and had the same experience. Over a couple years we went from a raucous, rough-and-tumble culture of open debate on the technical merits to a "polite" "respectful" org where nobody dared pop their head over the trench and bad decisions were allowed to continue unchallenged until they were nigh-irreversible. Much of it was due to a single executive's behavior. He was a big bully, always had to be right, and you knew that if you ever expressed an opinion about anything in his domain, he'd come after you like a fucking pitbull and worry your flesh until you submitted, using the power of his office and his connection to the founder to ensure you bent the knee.
I don't think polite and respectful are in opposition to open debate. If someone can't make a point in a debate without being polite and respectful, I really don't want to work with them.
A: We should use $FRAMEWORK for everything in $APP because it provides a better developer experience. Sure, long list views might be slower, but we hardly ever need to care about long lists.
B: Almost every screen in $APP is a long list! I don't think we should compromise performance on $FEATURE, $FEATURE, or $FEATURE for this -- benchmarks suggest they'd be four to five times slower.
A: [to B's manager, cc HR] By making such an obvious statement as "almost every screen in $APP is a long list" in a public Slack channel, B made me feel uncomfortable by implying I wasn't familiar with $APP, even though I've worked on it for several years. That is an unacceptably disrespectful way to hold a technical discussion.
Yep.
> The problem here is A, not B.
I'm pretty sure the implied problem is "B's manager" and "HR".
> The problem is _not_ the requirement to be polite and respectful.
The problem is that this requirement is not seperable from the details of how it's interpreted and enforced, and in a plurality of organizations that talk about "politeness" and "respect", it is in fact interpreted and enforced in a A(-hole)-ish manner, rather than a manner based on actual politeness and respect. Conversely, organizations that care about actual politeness and respect generally don't belabor the point as much as those that use it as a flimsy excuse, so people tend[0] to see a explicitly "'polite' 'respectful' org" as warning sign of the latter.
0: rightly or wrongly; I'm not sure how the statistics work out on that one.
TL;DR: as other people have pointed out, the scare quotes are there for a reason.
Obviously people should be polite and respectful, but when there is too much posturing around these facts, it's easy for malicious actors to weaponize.
[dang's pretty good about enforcing politeness and respect, so I won't try to present any examples. :-)]
polite: actual politeness
“polite”: unwilling to say something that another person might feel upset about.
————————————-
Also, there are ways that politeness obscures information-sharing. For example: I am saying the above at the risk of being condescending.
Nor do I, hence the scare quotes. In fact there was nothing polite or respectful about it—it was an environment of fear.
does not conflict with
"a "polite" "respectful" org"
while
"a big bully, always had to be right, and you knew that if you ever expressed an opinion about anything in his domain, he'd come after you like a fucking pitbull and worry your flesh until you submitted, using the power of his office and his connection to the founder to ensure you bent the knee"
has, in my experience more in common with not being the second. In fact, in my experience, what I think you mean by "rough-and-tumble culture" supports the pitbulls much more than any form of making good decisions.
"Polite" and "respectful" can be a form of bullying too. Ask the women in your friends & family if you don't believe me.
If you were to argue that the pre-fear environment was unprofessional, exclusionary, immature, and that it often led to miscommunications and hard feelings, I'd have a hard time disagreeing. It's certainly not my preferred style (any more) and I think we can all do better and be nicer without giving up the meeting of the minds that's so essential to progress, at any scale.
But what came after? It was polite—because nobody talked. It was respectful—because everybody was afraid. Afraid I can't go into further detail, but it led to far greater harm in the long run, for the whole world, not just the company.
But I still disagree that an appearance politeness and respect or attempt to promote them is either a sign of, or responsible for, harm to a company (or the world).
Oh, and I did remember a decent example: IBM in the 1990s up through 2005 (that being the last time I worked there). He who talked loudest wins. Anyone who could spew tech-sounding gibberish like a Star Trek screenwriter at high volume was apparently automatically made a Technical Fellow. It led to a long series of bad technical decisions---and anyone who disagreed was treated unprofessionally and immaturely, and excluded---that ultimately hollowed-out the company.
Is hurting your career if your are low level. If you are high level, it's hurting the team and the company. A new hire exec who thinks they know more than the veteran staff is a shortcut to destroying a company.
I've been places where such people have found their way to technical leadership. Finding ways to improve things can rapidly become exceptionally difficult.
At this point my only way forward is to have a personal blocklist of people whose technical leadership decisions I refuse to ever be subject to again. It's a shame, because some of them are also brilliant engineers.
Years ago I heard the phrase "set to bozo bit" to describe what you're calling a personal blocklist. It was described as an anti-pattern, and setting the bozo bit as a poor choice in nearly every case. It combines the bad parts of an ad hominem attack with the bad parts of personal arrogance and negativity. Don't think your other teammates won't realize you've "blocked" a teammate. Some of them will react badly, and potentially see you as arrogant.
I end with this quote from E.W. Dijkstra:
We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.
ACM Turing Lecture 1972, "The Humble Programmer"
You are saying this as a senior engineer who is in a better position to do something else about being on the receiving end of those personal attacks. I respect that and I hope you are able to keep doing it. But that isn't the case when the abuse is continuing over a long period of time, that means no other senior person stepped up to do anything about it, and the people on the receiving either have to quit, or risk getting fired for "insubordination" because they filed a complaint against a senior employee. Playing the office politics game and trying to avoid the abusive employee is not sustainable, as you've acknowledged.
That said, I have seen whole teams set the bozo bit on other parts of the organization. For example, I saw the head of an infrastructure group refuse to maintain or patch key services while insisting on ownership. The security organization set the bozo bit on this person and worked to protect the organization from the consequences of their choices. As an organizational tool, it can be a useful caution about known bad-faith actors.
As to your broader point, I think you're correct. Programming is difficult, error-prone, and thus benefits from an intellectual humility. Approaching it with the mindset that you are incapable of error makes this much more challenging.
Is this some close knit "paypal mafia" corner of the industry where people only work with people they already know?
All complex rewarding software jobs but it’s not the same as say, web development, in terms of choices and volume
I've found myself approached by recruiters for the exact large company and sizable department that one of these people has substantial rank in. It's already had an impact on my decisions.
When I interviewed for my current job, it turned out that I had separate, personal, previous-work-experience connections to all four of my interviewers and it only surfaced during each of the interviews.
And this is after changing my role/specialty.
----
It turns out that he was learning the ropes to transition to a huge leadership role in the company and no one knew it. He had previously led an entire division of a massive computer manufacturer, but 2008 wiped out most of his retirement and he had to come back to work. The very first thing that he did with his new responsibility once his role was announced was to give me a huge promotion to work directly under him with no interview.
To give a concrete example, at one point in my career, I was leading a team trapped on a death march project. We did not have the option of abandoning it, due to a meaningful fraction of the company’s revenue riding on its success. We did not have the option of more resources for the usual reasons. And we did not have the option of more time, because executives had already de facto announced when it would be done. Unfortunately, our direct upstream dependency in the company saw us as a rival - because they had wanted to own this project - and spent multiple years trying every way they could to sabotage and undermine us, including lying to us in meetings, lying about us to others, changing key system aspects to make our problem harder to solve, denying access to critical resources and people, pitting vendors against each other, and giving unsolicited negative peer reviews to people working on our team.
To say this was a difficult experience would be an understatement. I had four employees quit. I myself started having panic attacks twice a week and was in therapy for over a year to work through the crippling anxiety I was feeling every waking moment. I still have persistent health issues from the incredible stress of those years. We landed the thing, got our pats on the back, and then I quit.
The main guy responsible for this campaign of sabotage and mistreatment was much higher level than me in the company and punching down as hard as he could. He left successfully for a larger role at another company once it became clear that our team was going to hurdle any roadblock he threw at us. It was a knock-down, drag-out fight. I will never do anything like it again.
If I ever found myself working with that person or any of his leads again in any capacity I would quit instantly. The bit is set.
I also agree with the utility of the bozo bit. I worked somewhere where a long time engineer got Peter/Dilbert Principled at just the level of Team Lead. Once you've done four years or so in the org, they don't fire you just motivate you to quit. Anyway, this person gets moved into a Lead role with a team of one person where he can do the least harm. He's awful to work with, can't communicate properly and derails every meeting he's in with complete tangents.
Bozo bit!
And unless you live somewhere where new jobs are available that match your skills and current pay that can be enough.
> We did not have the option of abandoning it, due to a meaningful fraction of the company’s revenue riding on its success. We did not have the option of more resources for the usual reasons. And we did not have the option of more time, because executives had already de facto announced when it would be done.
And immediately concluded that the company has already committed to suicide and it's time to start sending resumes. Cross-team sabotage is icing on the cake but actually doesn't change anything here.
It’s hard to overstate how hard it is to leave this kind of project as a manager. You spend years of your life building relationships with engineers and trying (and sometimes failing, admittedly, but trying) to protect them. You know the situation is a disaster and you want to get out of it. But you’re afraid of letting down your people and hurting their careers. You’re afraid the next person won’t be able to protect them as well. You’re afraid of losing the years in your resume and not accomplishing anything. You’re afraid of being a failure if you give up. When your body is breaking down due to the stress, you’re afraid to lose your health insurance.
You’re right - I should have quit as soon as it was clear it was a death march. But in the shit, I found it almost impossible to lift my head up and say “this is literally killing me, I quit”. When each individual day is at your maximum trauma threshold, it’s hard to work up the time, willpower, or ability to interview prep and change companies.
I regret it immensely.
How would I not be doing the same thing if I looked at other people as needing to be "fixed" any time their thought processes don't work exactly like mine?
People who are Always Right and who believe that anyone who disagrees or sounds like they might be disagreeing is Always Wrong and must be shouted down... those people are not worth fixing. I assume these people are what GP is talking about.
And yes, those people are real. Quite rare, but real, and a major pain in the ass to route around.
Always assume that it is possible for you to be wrong. Even if you think you are not wrong... just consider what the world might look like if you were wrong, and you'll be fine.
I was mildly troubled by the comment towards the end, "There were third- and fourth-generation members who I had no idea about who had signed up to talk about reliability without being second-guessed to death and hounded by people who just wanted to talk about [...] "a lot of work went into that"."
I've been sensitized, I guess. Yes, the way we do things is probably stupid. Yes, your favorite new framework or method or language or whatever might trivially fix all the problems that you can see. Heck, you might even be the smartest person in the room. But, there might also be a reason things things are stuck at a historical, local maximum.
Inconveniently, sometimes the reason is "a lot of work went into that".
It's not personal. It's strictly business.
* i saw in one project shitting only on commits of a certain person, only on a technical basis, even if other commits were worse. i know this because of a real crappy testcommit i did. i did not get the shit the other person got.
* i still think, some swearings even for code are over the line, like linus torvalds retroactive abort of code, for example. We code and build things. many people poor their soul into this. and then other people come around and shit on it.
* people who shit are often very agressive and competent people. so less competent people may on the receiving end quite a bit, for example the guys who built the ubuntu base in that example. is their work doomed or shit? does this mean that they did not work well? are they bad workers or do generate bad results? should they be fired because their workresults do not perform well? can that be a general rule? and this line is not cut through by on-subject-shitters because "be an adult". and this in turn might create toxic culture. just as well as the toxic positivity which only comes more often because it's better understandable by non-it-guys from HR.
but i admit, at least it's another approach. we need more approaches and studies about them.
Though at least the former has the Geneva Conventions.
Humans are emotional creatures no matter the context.
You can't change that - all you can do is accept it then learn how to work with emotions (both yours and those of other people).
IMO, the "adults in the room" are usually the people who have deep technical knowledge and have also learned how to regulate their emotions. One without the other gets you nowhere.
During World War II, some Royal Navy battleships turned their anti-aircraft guns on their own torpedo bombers, because they were determined that carriers not get the credit for sinking the Bismark.
I'm an "older" engineer (just turned 59. I really don't give a damn what people think of it. That's one of the benefits of being my age). I have about 40 years of experience in tech; the vast majority, writing software. Sometimes, quite badly.
Also, almost my entire career has been in "shipping" software; not just "writing" it. This has meant that I've always been pretty damn close to the end-user; sometimes, to the point of receiving personal...erm...feedback. Let's call it "feedback." I've also been responsible for delivering complete product, with polish and support. It wasn't "someone else's job" to clean things up. It was my job (it still is -even more so, these days).
I've watched (and caused) some real Jurassic-scale disasters, over the years.
That's one of the reasons I write damn good software, these days. Nothing like being given a bag and a stick, and told to clean up the mess (even if it was someone else's mess), to teach humility and caution.
So, that means that I'm often the guy that says things like "Are you sure you want to do that? I did it once, and it did not end well."
In today's "cargo" culture, that's considered "being negative," or "being timid, and only the bold survive," etc. ad nauseam.
I'm told that I'm "not a team player," or that I'm "harshing the vibe" (I actually made up the second one, but you get the picture).
This is funny, because my entire life has been ferociously dedicated to making difficult things happen. Precisely the opposite of being "too cautious to leave the starting gate." I've always been about "That's a difficult thing, and here are some of the things that might go wrong, along with a bunch we can't foresee. Let's figure out how to do it, anyway. We just need to be careful and circumspect."
That's pretty typical with experienced folks in any profession; not just tech. We often have a lot of callous and scar tissue.
Yes, we can be so timid, that we are too gun-shy to try difficult things, but we are also fairly likely to actually make it happen, if we take on the job.
For myself, I tend to look for "dreamers," who have ideas, and then help them to actually realize their dreams, as opposed to some of these Krakatoa-level explosions that we see happen, because the execution of a perfectly good idea was flawed and ill-considered.
That's what "engineering" is to me.
Frame it up as "I'm just putting on DiBono's black hat here so that we can succeed, team" and suddenly you're a Thought Leader.
It should probably be https://...
Assume the best intent?
> This place hadn't quite ascended to the level of giving outages notable names ("Call the Cops", "Silent Night", "A Tree Did It" being just a few from elsewhere), it remained just "1152" after an associated ticket in a bug tracking system.
By the way, I also appreciated the post. I’ve been in similar situations in several occasions and never quite figured out how to avoid stigma and bad feelings for stepping on other peoples’ ego. I’ll give your technique a try
It's easier in a conversation to refer to an outage that was caused by a system called Maya by a descriptive canonical name like "Mayan Apocalypse", than to refer to it as 14351. As companies grow and accumulate more interesting outages, it becomes harder to keep all these magic numbers in your mental cache.
I always wanted to apologize to Sgt. Brink for having to take my calls that day.
I found myself in yet another tech gig in an attempt to provide reliability to a company that (as it turned out) was about to have its IPO.
Changing the subject mid-sentence does harm readability.
> I was talking about infrastructure, and specifically the wobbly mess of shitty "we're gonna run Flask everywhere and call it microservices" that somehow let people accomplish things over the Internet. Absolutely amazing.
If this is how she talks at work, it's unsurprising that people would find the attitude abrasive. (And if this is not how she talks at work, then maybe she could tone down on dramatization a bit? Well, I mean, sure, it's her personal blog so she can say whatever she wants, but still ...)
Hint: don’t tell me to smile more, either.
> What would happen is this: a couple of people would get to talking (on Slack, for that is what they used) about something technical. There might be a topic at hand, like "Ubuntu is doomed", and they'd be hashing it out, figuring out what that meant. Then, invariably, someone would pop in two or three hours later, hit the hated "start thread" button on one of the comments, and would start shitting all over them.
> "OMG why are you hating on Canonical" "Ubuntu is NOT DOOMED"
> And then the people involved would have to walk this person back and say, look friend, Ubuntu at this company is doomed, because the company has decided that everything is moving from flavor X to flavor Y, and all of the flavor Y images are built from Fedora (yeah, I know, ignore that for this story) instead of X's Ubuntu. So once we're done with the migration, Ubuntu at this company is a goner!
In a Slack channel, someone said “Ubuntu is doomed”. Not because Ubuntu, as a distro, will soon reach end-of-life, but because at $COMPANY, Ubuntu will be replaced by Fedora. That is to say, “Ubuntu is doomed (at $COMPANY)”.
What author is objecting to is people joining the conversation several hours later and expressing unhappiness that someone is predicting the end of Ubuntu everywhere.
But that’s because the new person wasn’t in the conversation, lacks context, and maybe should start by assuming the best intent rather than the worst intent.