A developer's guide to programatically overcome fear of failure
pagerduty.com
pagerduty.com
There are models for how to handle this stuff in other industries. What comes to mind is "Tiger Teams" and "EMS services". Both of these involve the deliberate creation of special organizational structures that respond to crisis. For both of them that means contingency planning and, importantly, practice. It's largely about training for things that might happen and thus being ready when they do happen-- even if they're not exactly what was expected.
Sadly, the model for crisis management in SWE (in most places) is just throwing around the word "accountability", putting people in the hot-seat, and hoping for the best. You end up with "hero-based" crisis management. In fiction, that works out great (James Bond types always win), but in reality you just end up with stressed-out people and flaky systems.
> What comes to mind is "Tiger Teams" and "EMS services"
I know EMS services of course, but what are Tiger Teams? A cursory Google search didn't turn up anything useful. Any pointers where I could learn more about them?
I have seen them at a linear accelerator facility and also in large factories. I think the first time the name was used was in the NASA space program (and looking at the wikipedia link someone provided that appears to be correct).
Basically, they are teams that are put together to deal with ongoing critical cross-cutting problems in the organization. They are, by nature, cross-functional. Large orgs tend to put talent into silos. Silo'd talent can make the org vulnerable to weird problems where no one feels enough agency to act using the excuse-- "it's not my job". A Tiger Team is intended to work across silos.
If they are smart, they have ways to roll back offending code.
But any company smaller than say a regional bank is going to have trouble affording this.
Now, if you using a cloud provider, you can design in rollback.
But of course there are surprises. A scenario I just thought up, when you discover on Black Friday that your order system can't handle more than 1048575 (that's 2 to the 20th minus 1, just to pick a slightly plausible number) orders in one day, and this limit is deep in your code and architecture.
I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here).
There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level).
I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".
I still meet SWEs who seem to believe they have no responsibility for any failures resulting from their code if it "worked for them in dev". No responsibility to design their code to resist failures or security issues in production, nor re-design it if the current design is failure-prone. No responsibility to performance test, functional test, or check with other teams on what a change might impact at scale. No need to come up with a rollback plan or test it.
If their product makes a ton of money, upper management enables this attitude, because nobody wants to kill the golden goose or risk losing its developers. It's a question of priorities. If the business does not require SWEs to care about risk, failure, reliability, etc, then they won't.
Someone who never makes mistakes or someone who deals with their mistakes openly and directly?
We know the first one doesn't exist. Some people like to make us believe that, but it's bullshit.
However, one big caveat: Owning mistakes, being open about them doesn't absolve us from making them, especially not in the future. We still _made_ the mistake. Should we dwell on that or get punished? Likely not. Severity, intention and so on matter too. But should we simply not feel bad about it? "Embrace mistakes"? Not so sure about that either.
When we say "embrace mistakes", what we really mean is "embrace learning". Mistakes are opportunities to learn for ourselves and others indirectly.
We shouldn't be victims of our bad feelings and beat ourselves up. I did that for years. It crushed my confidence and well-being into submission and years after I'm still recovering from that. But the solution is not to try and make us feel good about ourselves regardless. It's not _just_ a psychological problem.
Mistakes happen for a reason, even if they are as simple as "I didn't write that down". They happen all the time, we can't prevent that, but what we can do is decide what kind of mistakes we want to face and ultimately what kind of problems we want to solve, be able to solve - for us and others.
Don't get bogged down, don't beat yourself up and don't let others beat you up for your mistakes. But make yourself aware that you can actually learn from them and decide for yourself when you want or need to put the energy into doing that.
I get why people are saying things like "normalize mistakes" etc, living in some kind of police state environment that punishes every misstep is just miserable. Especially if you're working in software, where "no mistakes" is not a reasonable standard.
OTOH it shouldn't become an excuse for complacency, and especially not an excuse to tolerate incompetence and/or totally unforced errors.
He tells me a story about how he drove his truck under a low bridge, and did about $20,000 worth of damage to it.
When he got back, he was expecting to get fired (or worse).
Instead, his boss asked him what he'd learned from the mistake, and gave him his next assignment. When my friend asked (incredulously) why he didn't get fired, his boss answered "I just spent twenty thousand bucks, training you to be careful of clearance. You think I'm gonna toss that out?"
My friend says he learned a lot about management from these goombahs. They were managing very dangerous people, who were big investments, and couldn't be treated badly or disrespectfully.
I'm sure it didn't cost him too much. He used 20K as a nice, fat number, but they probably only needed to shell out a couple hundred bucks out of pocket. It was an easy act to do. Impressed my friend, but he didn't really think about the background.
They are like that. Not exactly The Sopranos. Quite different from TV.
This extends far beyond software programming. It applies to any form of self-development. I actually developed a habit formation program framework [0] built off of these principles (and a couple others, e.g aligned incentives, bias towards action, writing is thinking).
Here are examples of programs using it:
* https://themoai.org/work-life-balance
I don’t really subscribe to TDD, but they have a valid point: Start with something that doesn’t work, and work on it, until it works.
I tend to develop my software in a similar way, but I also start with a “fuzzy” design, which would give TDD people fits.
I fail every day. Repeatedly. It’s how I end up with some damn good software, at the end of the day.
I do have a blurb that discusses my design philosophy here: https://littlegreenviper.com/miscellany/evolutionary-design-...
I’m developing a fairly ambitious iOS app.
It’s a social media-like app, based on aggregate servers. I wrote the servers, myself, years ago. One is currently a world standard, and heavily in use. The other, was a server I wrote a few years ago as a “practice” project.
The app, itself, has been under development for over a year.
A big reason for that, is the team had only a vague idea of what it would do. It wasn’t really feasible to get them to hash out a more detailed plan, so I set up a “breadboard” system, based on Apple’s TestFlight beta-test system [0]. TestFlight is usually only engaged towards the very end of a project, but we’ve been using it since one month after my initial checkin.
TestFlight requires a fairly “complete” app, that runs without tripping any alarms (like crashing). Apple vets each release.
I can produce a fairly full-featured app, quite rapidly, so I was able to meet Apple’s quality bar, pretty quickly.
Because we are using TestFlight, we have some significant advantages:
1. The app is constantly at “beta” quality. Even with incomplete functionality, what’s there, works well. I cannot overstate the importance of this.
2. We can have non-technical stakeholders constantly “test driving” the UI, and providing relevant, useful feedback, on a continuous basis. They are also highly motivated and engaged.
3. Because Apple is constantly vetting the app, we know that we probably won’t have any nasty surprises, when we finally release.
During this year, we have explored a number of avenues that have resulted in “dead ends.” I’ve tossed out months’ worth of work.
That’s great. The best code I write, is the code I don’t write. I’m happy to tear out huge chunks of code.
The result is that we now have a clear, robust, ultra-high-quality app that is in the final stages of development. I still have a couple of areas of functionality to add, but we’re done prevaricating. We know where we’re going, and everyone is on board and psyched.
The quality of the app is also, quite frankly, jaw-dropping. We’re all thrilled to death.
Using TestFlight has allowed us to work through a lot of stuff that is often encountered during early MVP stages; without the brand damage and public humiliation.
If the TDD approach said "thank you" for the good parts (code structure, not breaking 500 other test cases), that would make the 13 errors less scary. One of the things I like about software is that it's "soft" - it should be possible to make changes without the fear of getting it wrong, and be able to fix them now while I see the problem, rather than procrastinating until later when I've forgotten about it.
The same "thank you" system applies when working with external contractors (e.g. language translations). We can assume they tried their best, and therefore might not know what else to say. Reporting 19 problems will get us a reputation for being nitpicky, hostile, even toxic. Rather, we should thank them for the 730 that they got right, and suggest improvements for the 19 that we think might be better.
With thankfulness and encouragement, TDD could really change the industry for the better.
I've always preferred carrots over sticks.
“When a person feels that he or she is becoming his or her undesired or feared self.”
I feel like a lot of my motivation to better myself and help improve my environment comes from that feeling. It's as if i have a very clear picture of who/what i dont want to become, but not so clear on what the "ideal me" would be.
Is this something that motivates HN as well or do you try to realize an ideal(?) version of you through your thoughts/actions
Many developers really dont care what happens in production, leaving a subset to stress out. Perhaps the solution is just be like the people who just expect problems and accept it.
That being said, here's another way to view it: of all of those deep seated fears you have, what do you know you can do to avoid certain issues and what can you actually not do?
Let's take the database for example. What's the deep seated fear here? Maybe:
> The disks crash, the data center bursts into flames and all the data is gone!
A solution to those is perform backups and get the backups off-prem. You have to explain the risk and justifications to the company as best as you can. Once you've done that, what is left?
Technically speaking, bugs in the backup script could exist, or maybe the backups get corrupted. Beyond that, there are the human ones. Maybe somebody else manages the script and breaks it, or maybe backups get unintentionally poisoned by malware getting into the backups. Or maybe even before that happens the company goes "eh, risk worth taking to save costs!" and you don't even get to implement the backups.
Here's the thing: these things are _not_ in your control at this point! They are out of your hands, and with humans like us who make mistakes like us. So problems are going to be expected, especially ones you don't realize right now. But they're not in your control! And yet, you did your part, and things will continue on.
These days, particularly in microservices, more and more developers are on call for their work, so they care more.
Still, I find a radical misalignment between my mentality, where you CANNOT make mistakes because you're working without a net, and development, where typically there are more layers of safety. I'm far, far more cautious and anxious about making changes and breaking things; they've historically been paid to produce. If I try to have a conversation about it, they're unable to grok that their threshold for the acceptability of making mistakes is much higher than mine is.
You NEED to be able to make mistakes in order to be able to produce, but it's a hard mentality shift.
"Systematically overcome your fear of failure" is a great principle, but why is PagerDuty of all companies promoting this? I can't help but think this is an attempt to market some kind of product, but I can't figure out what it is.
To drive traffic and increase sales of PagerDuty. It reads like pretty standard content marketing to me.
There is probably only going to be a retro for the incident if it was something really bad. Even then it's not going to focus on you, it'll be more about cross team efforts and higher level issues.
Having a personal tracker may help you understand your own weak spots, common sources of problems, and trends in your work. For each incident look at things like why it happened, what could have been done to avoid it, and the context that the mistake was made in (e.g. we had three issues going at the time and were under a product deadline for X which was due in the next two weeks).
To measure performance you could classify the mistake type and plot it over time. It could be something general like the preventable/complex/innovative buckets stated in the article with technical details like: missed the bug during PR review, misunderstood the system architecture, missed handling an edge case, etc. Having the issues clearly stated for yourself should help you not make a similar mistake going forward. If you have a trend in the data you can focus your personal development on areas where you're weak. Keeping a reference of issues that have happened should help with understanding the problem history a year later too. That could be good for planning future work since you'll know where some of the hard spots might be.
Errors are good given that they lead to learning. Frustration is fun.
Tony Hsieh from Zappos famously does one thing a day that scares him. At a certain point, potential failure is as scary as just trying something new.
https://www.nytimes.com/2021/01/26/technology/tony-hsieh-dea...
If you're weighing an option that opens you up to a higher risk of failure than other options, then depending on what type of person you are, you may have to evaluate whether you have enough of a support network or other "cushion" to absorb the mental blow of failure first. Failure is something that can be gotten used to, but before it is, you have no choice but to get past the damage to your mental health the failure causes, and some people are not capable of managing that on their own.
This is why I have to sometimes frame my decisions as: "Because I will fail considerably more often if I choose this path, I'm giving myself a considerable risk of becoming a miserable wreck incapable of doing the things I'm capable of now, or worse. So for the sake of preserving my mental stability, I will have to do something else."
DIY is great, and an outside professional perspective can also super charge your efforts and put you years ahead.
Once I unlocked this, the world became flat. I didn't care about the journey, just the goal. Whip something up. Learn how to socially manipulate expectations. Make it seem like you work. Half ass something and convince people there were significant difficulties.
None of your code matters. You are paid to support. Not to reinvent the wheel. So support it with the easiest code possible. Quit trying to do something fancy. It doesn't help when layoffs come about. What does is cozying up to VP's and management who know you work and get things done. Your portfolio means fuck all if not a single person knows what you do, even if you are a better programmer.
I think we mostly don't love winning or succeeding per se, even when that happens it's not the joy of winning and succeeding as much as it is the relief of not having crashed and burned.
The whole mental process is dominated by loss aversion, not by victory enthusiasm.