Troubleshooting: A skill that never goes obsolete
autodidacts.io
autodidacts.io
There's a certain way of reasoning about the problem and thinking about what it might be with limited information in a systemic way. It's also a bit broader than debugging - you can do a lot of troubleshooting (sometimes faster and more effectively) by doing things other than reading the code.
It's also been somewhat of a career advantage because it seems to be both more uncommon than standard dev for someone to be really good at and something that most people dislike (while it's my favorite thing to do). It also overlaps a lot with other more general types of problem solving.
Anyway - a lot of the article resonates with how I think about it too.
You build credibility by jumping in and doing a lot of support type stuff early on (which then also makes you better at whatever the product is, more familiar with what sucks for users).
In my company, I'm often the person who joins a production bug troubleshooting call, after sometimes hours of investigation, and rapidly identifies the root cause.
My typical workflow is:
* Clarify the issue and our assumptions. Often, simply restating the observed behavior aligns everyone.
* Pose questions to validate or challenge those assumptions.
* Suggest alternative methods to test the primary hypothesis.
Often, testing the initial hypothesis reveals its inaccuracy, leading to a swift discovery of the actual root cause.
Ultimately, it comes down to critical thinking and questioning assumptions I think.
1. You have to start having hypotheses that you test, but should be ready to throw them away as quickly as you thought of them, when the results from testing it says so. Let data
2. You should preferably think hard about effective way to quickly rule out influencing variables and so quickly square in on the area where the erroneous effect is coming from.
3. You have to really rule out confounders. Make sure to turn off any caches or similar that might play games with you.
The area where I see most colleagues fail in this process is not being stringent enough with things like ruling out confounders and being systematic about organizing the outputs from hypothesis testing, to make sure you are 100% which outputs belong to which inputs etc.
It is the discipline and strictness in the process that will do the trick. Anything less and you will just trick yourself.
The other thing I see trip people up is being unwilling to make a fast hypothesis that can be easily tested to narrow scope.
Instead they’ll often try to look at the code to understand but that’s usually slower for anything remotely complex.
That's not good. The problem with troubleshooting is that it messes up with your reward system. After you fix a hard-to-debug problem, you feel a sense of accomplishment. Which would be ok, but the problem is that this sense of accomplishment is often time higher than it should be. You go home at the end of the day thinking "well, today I didn't build anything, but it's fine, because I fixed that bug". You are becoming complacent.
If you end up saying to yourself, like the author of this blog here, that you troubleshoot more than you build or you do, then you have a problem. Soon you'll be seen by others as a car mechanic. Maybe a reliable car mechanic. But reliable car mechanics don't get paid a lot.
This might be a controversial take but here it is: being proud of your troubleshooting skills sits somewhere between being proud of your typing speed and being proud of your word document formatting skills. These things never go obsolete, but don't fool yourself into thinking they are gold currency on the job market.
It's a tough balancing to make sure you sell yourself correctly and fight to work on things you want to!
To prove the point we put him on a strategic rewrite and gave him master/trunk while the entire team moved to a feature branch for 6 months. This was complimentary to his ego as he was sick of us bureaucrats in the rest of the team telling him what to do and being such a burden on his genius creativity.
By the end he was unable to build / run his own branch, while the remaining team lost no velocity and was making regular releases to end users. The choice was easy at that point.
Compensation has been decent with this approach over the years. I could have made more staying longer in a darwinian bigco but the work was not fulfilling.
By far the easiest way to do so will be to find another job. If you can't do this, yea, mentality will lock you in to positions you don't want to be in.
Fixing things only gets notices by coworkers and good managers.
No.
Building new things is sexy and highly visible. It's easy to say about yourself "I built that cool new feature" or better, to promote "I might build something of incredible value". You're front-and-center with decision makers, shaping the future.
Conversely, debugging is perceived as a cost center. "I fixed that critical infrastructure that used to work with only minor interruption" or "Maybe everything will break and you'll need me to fix it" are not nearly as exciting. Worse, the best maintenance is completely invisible, fixing problems before they are felt. You're in the background, dealing with the legacy of the past.
I'm a troubleshooter. I fix problems. I keep my head straight in a crisis. Every job I've had across 3 decades, regardless of my actual title or formal responsibilities, I'm the firefighter. People call me when they can't figure something out. People call me when something big breaks and needs to be fixed urgently. Even if I'm not an expert in the broken thing, they call me in. They call me because the experts are often floundering and not making any progress because they can't troubleshoot their way out of a wet paper bag.
I do not feel this has held me back professionally. I have been loved by management and peers in all of these jobs. When I nearly left a prior employer because much of the work wasn't aligned with what I wanted to do, management created a new role with better aligned work and higher pay to convince me to stay. In my current role, I'm very happy with my salary, working environment, management, and team.
I wish troubleshooting skills were as common as typing and document formatting skills. I wouldn't need to help out nearly as many people because they could handle their own crises.
If only your experience was universal in that regard! I once had that role in an early-career job -- but I was looked down upon by peers and management because I was doing mostly maintenance work. The "good" developers, in their minds, were the ones shipping the most new features -- the irony being that those features would then blow up out in the field, at which time they landed on my desk to turn them into production-worthy code.
>Yeah boss I can fix it, but how much is it worth to you since this isn't in my job description.
This describes a sizable portion of my career. It's lucrative, it's gratifying, and it's fun. It's as close as I'm going to get to being a "kick-ass mercenary".
Seeing new environments, new applications, and new problems never gets old. The stories that come from the work are priceless, too.
> I wish troubleshooting skills were as common as typing and document formatting skills.
When I conduct interviews this is the main skill I screen for. I think it can be taught, but somebody who already has it and is missing some particular technical experience is vastly more valuable.
A good troubleshooter can enable higher output across a team because they are like grease in the machine. Particularly indifferent troubleshooters become a net drag because instead of being able to help others they are always interrupting others for help.
Ever notice people get more stubborn and stuck in their ways over time?
It's possible you cannot teach people to want different things.
I think you can teach someone to troubleshoot in a procedural and methodical manner, but they will always lack the creative "spark" that comes from being actually interested. Procedural troubleshooters are useful, but they won't exceed the bounds of the model they've been taught to work under.
Also, for example, so many developers could see an issue and fix it without really understanding how the fix works.
I literally cannot live with that and have to understand why something works the way it works
I was a manager for over 25 years, and this was exactly the type of thing that I looked for.
LeetCode tests actually tend to bias against that kind of skill.
But she didn’t panic, she cracked open the debugger and went section by section through the code until she finally spotted her typo. Which is exactly the sort of person who won’t crumble every time something doesn’t work exactly the way the documentation says it does. We hired six people, and only renewed two, of which she was one. So as far as I’m concerned, I succeeded in my interview.
> I wouldn't need to help out nearly as many people because they could handle their own crises.
They don't need to, because there's always you who can figure out boring minutiae for them while they deliver business value.
Not everyone is cut out to do that. Asking them to look at a puzzle derails their entire day, almost every time instead of just when it’s hard. So even when it’s their puzzle they resist picking it up because it’s a guaranteed bad day.
I must have gotten lucky then because I’ve built most of my career off my incredible troubleshooting capacity and my communication capability.
I’m not earning Silicon Valley money (because thankfully I don’t live there) but I’m at the top end for salaries in my country. I out earn the vast majority of devs/devops people I know.
Maybe it’s a bit unfair though because I troubleshoot at a very different level to most I suspect.
edit
Or maybe I’ve been applying the wrong word to what I do my entire career…or misunderstanding why I’m valued. This is more possible than it might seem.
edit
For further context, i've held a wide variety of positions now within IT, including executive management. The problem is still, and I know this will sound stupid, I don't know how it is I got here, or why people valued me enough to keep me promoting me. I never even asked for the promotions they just...happened. This has made applying for new jobs harder, because I've never actually been entirely sure what my value is, so I often take jobs that are lower than my last job, but then everytime its not long before I end up well above that again.
I eventually assumed it must be my troubleshooting capacity. I asked a CEO I worked with once at a smaller startup why it is he kept promoting me, and I got this story about how by just being in the room, everyone around me wants to do better work. Not because they are being told to do so, but because the work I do apparently just inspires people around me to do better. I was the truest example he had seen apparently of 'lead by example.'
It's been very problematic because despite earning good money, and having never struggled to find, retain or advance in a job, I still don't truely know what it is exactly i'm good at.
I think im terrible at explaining this. Every time I have tried to talk to someone about it in real life they just end up telling me I have impostor syndrome. Of course I do, I don't really know what the fuck it is I do.
I dunno, the mechanic I go to is reliable and so busy it's hard to get a slot these days, and he seems to be doing very well for himself. So many mechanics are unreliable
Obligatory Kernighan’s law: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
this argues for a messed up reward system for the company, not the engineer. all sufficiently large systems have bugs and performance issues, and adding a measure of reliability, stability, and speed is just as important as adding features.
"Good troubleshooter" might not look great on a CV, but all of your coworkers naming you as the most valuable member of the team, and a natural leader, is worth more than any feature launches.
"I found the root cause and corrected it. It may be an issue in these other two places so we should check there as well."
"Great. How do we avoid issues like this in the future?"
"By doing X thing a different way, and ensuring that Y thing is also in place."
I stopped trying to be that person because it came with too many costs. It wasn't that I didn't want to do it but that I wanted other people to be able to in my absence.
Making troubleshooting skills a profession in itself makes reliability a property of a specific person or team and not a property of the system. The former doesn't scale.
If you're a wizbang mechanic working for an import car repair, you can get paid a lot more in hourly labor fees than some of us make.
Personally I've seen people who struggled on a trivial problem get a strong rating from their manager afterwards.
> reliable car mechanics don't get paid a lot.
Hell, I've seen some that are so busy that they won't take new customers!
But I don't want to be known as the great troubleshooter in my team. I want to be known (and I am) as the guy who builds stuff.
I think you're right in some cases (when working in a field one has mastered, for example), and I think I could probably go in the direction of getting it right the first time.
But the way I see it, any time I'm doing something new or innovative, I'm doing something I don't know how to do, which takes trial and error; and troubleshooting is basically figuring things out by trial and error, in a systematic way.
Though a lot of time it is used for fixing bugs, I think troubleshooting as a skill and mindset is equally useful for creating new things, where you are solving for something.
Then it was placed in news.ycombinator.com/pool (https://news.ycombinator.com/pool?next=43176091), and got two comments; credit_guy's comment was one of them.
Then, today, it hit the frontpage.
Notice that if you mouse over "9 hours ago" on the story it shows the timestamp 2025-02-25. 9 hours ago was not 2025-02-25. If you mouse over the "7 hours ago" on credit_guy's comment, the timestamp shows 2025-02-26. One day after it was submitted, two days before it made the frontpage.
I honestly don't know: do they not? The reliable mechanics in my city seem to do tons of business and charge significantly higher prices than the competition.
I can think of a lot of software that I have stopped paying for because they did not fix bugs or performance issues. I can think of far less software that I stopped paying for because although it was reliable, they did not add more features to it - but I am an individual, not a business, and likely am not representative of the average software-buying individual.
In software and systems that I have built for myself, the impact of fixing a bothersome bug is usually far higher than adding a new capability, but I may just be more bothered by bugs than most. Reliability and smooth, predictable operation are very important to me.
Not a great analogy. Reliable car mechanics often get paid very well in comparison to their peers. Used to be one. Got into tech as a result of how much tech got into cars. Do they pay as well as tech jobs? Depends. I made more money and worked less hours than my buddy in IT at one of the largest corporations in America in the same city (granted not a techhub like SV, Seattle, NYC, etc, but most places aren't).
The key differentiator here is not time spent building vs time spent repairing (troubleshooting). It's knowing what's worth spending your time on, and when to say "No", because not everything needs fixed, nor is every problem necessarily yours to solve.
Truly good diagnostics skills is knowing what's worth spending time on, regardless of whether it's repairing something that exists or building something that doesn't. A tire with only 10% of treadwear could technically be replaced with something better, but is that worth anyone's time or money? Probably not. But if the tire on the opposite side is still brand new, and they were replaced at the same time, diagnostics tells you the alignment is off, and that issue - whatever it may be - very well could be worth everyone's time and money to fix.
Code is no different. Don't try to fix/improve/build everything. Focus on what matters. Good troubleshooting/diagnostic skills is a big part of knowing what does, and doesn't.
You can feel good about addressing risks, the opposite of complacency.
I think that's just having a job. There's nothing inherently better about building than fixing. Hell, anyone can build something these days. You can get chatgpt to write you a fully functional bootloader without knowing a single bit of assembly or how booting operates. Being able to grasp and fix things is already the superlative talent worth hiring for.
> But reliable car mechanics don't get paid a lot.
The equivalent in our industry is worth a lot more money than someone who can only build. I think "building" is a lot closer to your analogy of using a word processor than "fixing" things is and you've got the reputation of the two skills completely swapped.
I think of it as "offensive" and "defensive" building, ideally you want to be on the offensive (i.e. building stuff that wasn't there yesterday) but you have to balance it with good defence (i.e. adding a certain type of anti-fragility to your system by fixing bugs due to your features being exposed to the real world).
Saying this, I've never met a good engineer who wasn't very good at troubleshooting so perhaps it's more of a consequence of building than a skillset.
There's value in fixing things in the moment and then feeding them back to your engineer and architecture functions to address endemic issues so that everyone benefits.
So many issues turn out to be the smallest configuration mishaps, this is why I also promote using a debugger as much as I can as well - there's nothing quite like being able to see _exactly_ what's going on.
I don't know, the speed of just reading code and maybe inserting some diagnostic messages is hard to beat. It's a pretty bad day if I feel like I need to bust out a debugger—99% of "seeing _exactly_ what's going on" is not going to be relevant and will just distract you.
Basically the only time I pop open the debugger is when it's otherwise difficult to see runtime behavior—say, a certain condition in a server for which you can't easily access logs. Outside of that it feels like major overhead and distraction from getting the bug fixed. Plus iterating without print statements is a tedious, tedious affair.
Don't get me wrong, it's a critical and necessary skill that junior devs often struggle to understand and master. I just think over reliance on the debugger will slow your velocity over time when most bugs have straightforward causes easiest to see by simply reading the code (which you'll have to do anyway with a debugger). I can't tell you how many times I've had to tell devs to put away their tools so we can calmly analyze the code without flipping back and forth between views. The vast majority of the time it's the second pair of eyes that resolves the problem.
C++ is especially hard to reason about, though. The cognitive overhead of "running the compiler" in your brain is just staggering. Even scala doesn't feel that bad. Java code feels like building with LEGO in comparison—super, super easy to read and process.
Just finished the book recently. It's very insightful. As someone who considers himself good at debugging* and is still trying to improve debugging skills and efficiency, I view this book (and similar resources like this article) as a guide that also helps me reflect on what could have been done in a better way next time.
* Multiple times, I helped others find the root cause of a bug after they spend hours at it and have no clue what is happening
I believe Hillel Wayne gives a copy to every junior dev he meets.
I think it's "Wanting the right thing" (This includes figuring out what the right thing is) and "Being able to articulate your wish clearly" (This includes clarifying your thoughts).
You may still wish to not fix it for various reasons.
Yes, debugging is important, and too many people can't do it, which is unsettling considering how many bugs those people are putting into the code in the first place.
Or debugging and understanding the reason why a system isn’t behaving as expected. And pinpointing the part of the code that causes the behaviour that is not desired.
In another field—In IT Service Management (ITSM)—there is the distinction between incidents and problems. If you see many incidents coming in that are related, you sit down and start doing a root-cause analysis, basically a form of debugging. Or troubleshooting.
So yes, this is a skill that is timeless.
Shouldn't this say "... than when in a hurry."?
Don't wait for stuff to break and react. Be proactive and find ways to demonstrate how it can break and how to fix it.
Wikiquote cites the book “Cosmos” page 218 as the source of this cool quote. https://lccn.loc.gov/80005286
Exporting telemetry events with a wide set of attributes to an observability platform is a great approach which can provide an extensible way to expose additional information about the events.
That really helped me to become a good troubleshooter.
There are problem areas where it is a lot easier to assume everything is a 10/10 monster.
If you start every journey with "power cycle the device" and always wind up with a bridge call between 3 vendors, you might as well get the bridge warmed up the moment something throws a warning.
Oftentimes, getting someone on the phone can be a bit of a circus act regardless of what the contracts say. Over reacting early on can minimize total time to resolution.
We'll get paid peanuts for it, but hey, we should be thankful for the work in the first place!