Hacker News Reader

Comment by qz2 | Hacker News Reader

I disagree.

Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make.

What happens is we have growing complacency and disassociation from consequences.

Do you press the button on something potentially destructive because your are confident it is ok through analysis, good design and testing or confidence it is ok through trite complacency?

The industry is mostly the latter and it has to stop. And the first thing is calling bad processes, bad software and stupidity out for what it is.

Honestly these guys did good but most will try and hide this sort of fuck up or explain it away with weasel words.

View on HN

You should have zero fear instilled when pressing any button. The system or process has failed if a single person pressing a button can bring something down unintended. Fix the system/process, don’t “instill fear” onto the person, it’s toxic, plus now you have to make sure any new person on boarded has “fear instilled”, and that’s just hugely unproductive

That’s precisely my point. A lot of people have no fear because they’re complacent or ignorant rather than the button is well engineered.

But to get there you need to fear the bad outcomes.

I’m sorry, but this really hasn’t been my experience at all in web technology or managing on-prem systems either.

I used to be extremely fearful of making mistakes, and used to work in a zero-tolerance fear culture. My experience and the experience of my teammates on the DevOps team? We did everything manually and slowly because we couldn’t see past our own fear to think creatively on how to automate away errors. And yes, we still made errors doing everything carefully, with a second set of eyes, late night deployments, backups taken, etc.

Once our leadership at the time changed to espouse a culture of learning from a mistake and not being afraid to make one as long as you can recover and improve something, we reduced a lot of risk and actually automated away a lot of errors we typically made which were caused initially by fear culture.

Just my two cents.

I’m not talking about fear culture. It’s a more personal thing. Call it risk management culture if that helps which is the inverse.

Manual is wrong for a start. That would increase the probability of making a mistake and thus increase risk for example. The mitigations are automation, review and testing.

I agree with you. Perhaps fear was the wrong term. I treat it as my personal guide to how uneasy i feel about something on the quality front.

I recalled Akimov pressing the AZ-5 button in Chernobyl...

"fear required when working out what decision to make"

People like you keep making the same mistake, creating companies/organisations/industries/societies that run on fear of failure. We've tried it a thousand times, and it never works.

You can't solve crime by making all punishments hearsh death, we've tried that in 1700 in Britain and crimerate was sky high.

This culture gave us disasters in USSR and famine in China.

The only thing that can solve this problem is structural change.

I think my point is probably being misunderstood and that is my fault for explaining it poorly. See I fucked up :)

The fear I speak of is a personal barrier which is lacking in a lot of people. They can sleep quite happily at night knowing they did a shitty job and it's going to explode down the line. It's not their problem. They don't care.

I can't do that. Even if there are no direct consequences for me.

This is not because I am threatened but because I have some personal and professionals standards.

HN likes to downplay this, apparantly, but not everything can be boiled down to bureaucracy.

Yes, medical professionals use checklists. They also have a harsh and very unforgiving culture that fosters craftsmanship and values professionalism above all else. You see this in other high-stakes professions too.

You cannot just take the checklist and ignore the relentless focus on quality, the feelings of personal failure and countless hours honing and caring for the craft.

Developers are notorious for being lazy AF, so it's not hard to explain our obsession with "just fix the system". It's a required but not sufficient condition.

'The system' includes the attitudes of developers and people that pay them.

Everyone takes job of a medical proffesional seriously, from the education to the hospitals that enmloys them to the lawmakers to the patients.

When you pick a surgeon, you avoid the ones that killed people. Do you avoid developers that introduce bugs? We don't even keep track of that!

You can have the license taken away as a surgeon, I've never heard of anyone becoming unemployable as a developer.

You are not gonna get an equivalent outcome even if tomorrow all developers show up to work with an attitude of a heart surgeon.

However if suddenly all data loss and data breaches would result in massive compensation, and if slow and buggy software resulted in real lawsuits, you would see the results very quickly.

Basically same issues as in trading securities: no accountability for either developers or the decision makers.

> medical professionals

operate in an environment where they don’t fully understand the systems they’re working with (human biology still has many unknowns), and many mistakes are irreversible.

If you look at the worst performing IT departments, they suffer from the same problems: they don’t fully understand how their systems work, and they lack easy ways to reverse changes.

> The only thing that can solve this problem is structural change.

Well, care to elaborate on this? What do we have to change, and to what end?

You're speaking to the mistake. The comment you're replying to is speaking to the write-up analysing the mistake.

Blog posts analysing real-world mistakes should not be met with beratement.

Most will hide it away because being truthful will hurt current business or future career prospects because people like yourself exist who want everyone shitting themselves at the prospect of being honest.

In a blame free environment you find the underlying issue and fix it. In a blame full environment you cover up the mistake to avoid being fired and some other person does it again later down the line

No.

There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another.

I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome.

If I fuck up I’m the first person to put my hand up. Call it responsibility culture.

See that leading with a "No." there

That's what we're talking about. I hope you don't have direct reports.

Next time be honest "Just shut the conversation down, everyone's a dumbass, I'm right, you're dumb" it'll be quicker than all this back and forth actually trying to get to a stable middle ground :)

That's a tad ironic is it not?

All I am calling for is people to take responsibility.

What is the future value in that ? From a system and reliability point of view ? Genuine question - not trying to be a dk

The point is that if you take responsibility then you're taking pride in your work, are invested in it and willing to invest in self-improvement and introspection rather than doing the minimum to get to a minimum viable solution. The outcome of this is an increase in quality and a decrease in risk.

Wow - "...invest in self-improvement and introspection.."

I would hate for that to be our system reliability improvement methodology.

Ok fine now I'm being slightly a "dk" but really ?

Well you can also apply a QMS if you want but all that does is generate paperwork full of accepted risks...

I think you missing point, I love the idea about "bringing aviation methodology" to lower error/f-up-rates" for the software industry.

No one is not saying don't take responsibility, they are saying - as I understood it:

Have a "systematic-approach" to the problem, the current system for preventing "drunk pilots or the wiping of production db's are not sufficient" - improve the system ! ! All the "taking responsibilities and "falling on one's own sword" won't improve the process for the future.

If we take the example of the Space-Industry where having 3x Backups Systems are common (like life support)

It seems some people's view in the comments stream is:

"No bugger that, the life-support systems engineers and co should just 'take responsibility' and produce flawless products. No need for this 3 x backups systems"

The "system" approach is that there is x-rates of failures by having 2 backups we have now reduce the possibility of error by y amount.

Or in the case of production-dbs:

If I were the CEO and the following happens:

CTO-1: "Worker A, has deleted the production DB, I've scolded him, he is sorry and got dock a months pay and is feeling quite bad and has taken full responsibility for his stupid action this probably won't happen again !"

CTO-2: "Worker A, has deleted the production DB, We Identified that our process/system for allowing dev-machines to access production db's was a terrible idea and oversight, we now have measures abc in place to prevent that in the future"

I'd go with CTO-2 EVERY day of the week !

Yes. CTO-2 is my approach. As the CTO I fucked up because I didn't have that process in place to start with. To buck stops at me.

CTO-2 also has the responsibility of making sure that everyone is educated on this issue and can communicate those issues and worries (fears) to his/her level effectively because prevention is better than cure. Which is my other point.

Exactly right.

The "comfort" will come from taking responsibility and owning and correcting the problem such that you have confidence it won't happen again.

Platitudes to make someone feel better without action helps nobody.

The fear is a valuable feedback mechanism and shouldn't been ignored. It's there to remind you of the potential consequences of a careless fuckup.

Lots here misunderstood this I think.. clearly the point is not to berate people for making mistakes or to foster a "fear culture" insofar as fear of personal attack but rather to not ignore the internal/personal fear of fucking up because you give a shit.

Reply on news.ycombinator.com