Hacker News Reader

> One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.

owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.

Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.

Something we tell employees during our onboarding in a technical comes down to this.

- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.

"Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky"

Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.

Like the engineers who never have to refactor, because their code is good enough in the first go.

The only way to guaranteed never have mistakes in your code, is to not have any code at all.

To be fair sometimes people hyper focus on refactoring rather than fixing business problems, which isnt really helpful either, have to find a decent middle ground.

You are absolutely right, I forgot about the other extreme.

> owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.

If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).

Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.

2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.

In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.

My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.

Man that sucks. You deserve better for jumping in and helping. Hope you're in a much better company now.

Leaving that company was one of the best things that happened to me. It also thought me that when a company, especially a big one, blames a low level employee for a big mistake or some visible incident, there is something very wrong there. There is usually so much politics, so many layers of management, that blaming one person that does some actual work is the easiest way of hiding your actual problems.

in my experience this is detrimental for a company in the long term. It results in people who are in charge of taking leadership not actually leading the company and it shows everyone in the company taking risks is instant failure in their eyes.

To give you a counterexample.

The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.

Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.

Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).

The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.

In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.

Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.

Great for you. The company that I currently work hired a new manager, he is amazing on how he shows that we are a team and a "person" mistake, is actually, the process which need better thinking/layout.

> Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.

Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.

If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.

I've also found that being forthright about your mistakes gives credence to your _disavowal_ of responsibility. When it's not your fault and everybody's looking for a root cause, if you're known as someone who is quick to admit fault, then when you deny it everybody accepts it.

I was on the other side of that Netflix outage, as the main contact point between Netflix and AWS at the time. Amazon did give us a detailed rundown of what happened, but was very specific to not name names, nor did we ask. We all agreed it was an excellent learning opportunity for both AWS and Netflix.

That outage is what drove us to rearchitect all of Netflix to be multi-region.

The bottom line is, the PR damage done by SolarWinds' CEO to his company stands out the most, and he can and should fix it.

The CEO blaming an intern is the big thing that looks bad, but it's also very telling of their culture and how they got in this situation in the first place.

The CEO saying this, they are inadvertently signaling that they want a certain class of customer, a customer who agrees with and would make similar statements.

Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.

SolarWinds is looking for an Equifax not a Netflix.

What’s left unsaid is that not just the CEO, but every manager down the line has made it clear that it’s okay to toss the blame down the line.

Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.

The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.

I am deeply ambivalent about my tenure at Amazon overall, but I unconditionally applaud the level of psychological safety in their dev culture: there is a shared understanding that any system failure caused by a single person is necessarily a failure of the process that governs changes to that system.

If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.

My experience there was overall very negative. My comment above, though, was actually a very bright spot, and I want to call out the positives that exist.

At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.

Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.

At well managed big tech companies, stepping on a landmine like that is not only not detrimental to your career, but it can actually help you (unless you didn’t do that by mistake, but by being reckless and on purpose skipping all the protection layers). You can own your accident and, depending on the complexity, create whole teams to properly fix it, so no one else can cause an outrage like that.

My colleague typically asks this question in interviews, and I've occasionally borrowed it from him: tell me about the last problem you caused and how it got resolved. It's one of those questions designed to just get them talking and there aren't too many wrong answers, except maybe, you know, not being able to think of anything.