Hacker News Reader

Think of what you're saying on a meta-level here. You would basically be telling this guy, "I fucked up, but I'm blaming you for treating me like an adult instead of a child. You screwed up by giving me enough autonomy that I could make a mistake." That might not exactly be a smart career move..

On a more personal level, if someone shows you kindness in the face of a mistake, the last thing you want to do is throw it back in their face and go on a tirade against them. That's a very quick way to make sure nobody ever wants to work with you again.

Good point. It's important to take responsibility even if you think there should have been guardrails.

Reminds me of an old VP of mine who came thundering out of his office, absolutely livid, shouting, "What idiot gave me permissions to delete the source backups???"

From a security perspective, there is the least privilege concept.

Why does the VP have that kind of access by default? I understand having a separate account if needs are there, of having some sort of priv-escalation.

In my new job as network engineer, all the RH boxes have SElinux off. I don't get it, nor can anyone give a cogent answer. And up 4 floors, we have a selinux kernel Dev.

I'd greatly like to even limit roots potential damage against users' data (in this case, DBs).

> Why does the VP have that kind of access by default?

Two common reasons. First, because the VP or someone in a similar position resented not having such access; it's sometimes hard for people to accept that people who work "for" them hierarchically have permissions they don't. And second, because either they or the person who previously held their position had a legitimate reason for such access in the past, and didn't drop it when they no longer had such a reason.

But yes, even if they have a legitimate reason for such access, they shouldn't have that permission all the time, only when they intentionally don that hat.

I disagree. If you have something that is of significant value to the company, you need to hedge your risk. Automation is a particular hedge. People fail, and while processes do too, they often fail less.

Would you be comfortable moving something fragile by hand that is worth a lot of money? Maybe. Would you prefer if the fragile item being moved was done with an automated process that was shown to best protect fragile items? I would.

"I'm really sorry I caused that car accident." "That's okay, next time take Driver's Ed and don't drive on sidewalks." "Thanks, I'll make sure to do that... by the way, why can I even drive on the sidewalk in the first place? Why wasn't it required to take Driver's Ed?"

I'm very obviously using hyperbole here, but questioning a process, to me, often shows a level of maturity in engineering. It's not blame-shifting - you clearly still made the mistake - but it's fixing a root cause.

I do think it has to be brought up carefully and with the right tone at the right time though.

I think you've hit the nail on the head at the end with "right tone at the right time" - the right time is almost certainly not when one has screwed up.

Or if you think it can and should be fixed at that institutional level tell them what you've learned to do better on a personal level and then volunteer to help lead the effort to create the automation/systems to prevent anyone else from making the same mistake in the future.

Well, "process" and bureaucracy tend to march hand in hand. I think it's easy to get carried away with creating "processes" (i.e., additional bureaucracy) when sometimes people just need to show good judgement. Pushing code right before you leave for the weekend is just bad judgement, and the guy learned his lesson. Heavily bureaucratic institutions laden with processes are not exactly known for their efficiency or pleasant work force.

OTOH, at the most sophisticated organizations, being afraid to push on Friday demonstrates an embarrassing lack of QA infrastructure.

I read what he was saying differently. When errors like this occur it's very much worth evaluating what could be done to avoid it in the future beside me just learning my lesson. There are many general team behaviors (e.g. Automated functional testing, automated load testing, static analysis, code reviews, non-production environments, etc.) and things specific for the problem at hand (what test is missing that would have caught this?) that must be considered after a major issue. If these aren't evaluated then there it is likely they will happen again. I don't believe these are being treated like a child. Instead these are the team deciding to be adults by avoiding emergencies and being more confident when releasing.

Why not both? Of course you can accept responsibility for this mistake but then suggest creating a testing suite to run commits through before making them live?

Yip, that's exactly what I did today. I made a mistake which borked layout on one page. I thought I'd looked at it, didn't, and it got deployed. I'm now suggesting the company use continuous integration and some automated screenshot type tests for catching these issues.

I hate making mistakes, but if more robust procedures are the result, well, it's an overall win!

I worked for a boss once who essentially engineered this kind of problem to exist: he knew not having automated testing etc. was a problem, and instead of asking his (too junior) programmers to build up a test suite, he decided to let them fail and discover for themselves why writing up test suites is a good idea.

His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen. They have to experience hunger and ask to be taught to fish. If people don't have the soul-crushing experience that teaches them why something is really important, they never really internalize why it's important. That the reason why we appreciate that this stuff is so important is BECAUSE we suffered through something that taught us that it was important, and that if we don't give junior folks that same kind of experience, then good practice just becomes something on the list of priorities and not a moral imperative.

There's something to that, but at the same time, it's basically a defense of institutional hazing on the investor's dime. So take it for what it is.

> His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen.

I think you can, it's just not nearly as efficient. College and trade schools are essentially about mixing a very small amount of actual necessity (deadlines, tests, etc) with copious amounts of being told what's best and how to accomplish something. This works well because it's both more palatable to a wider audience of people, and it accounts for people's different interpretation of events. There may be multiple things to learn from any particular failure, and without some guidance you may come away having learned few or none of them the first time. Being told what to expect before hand, or what you could have done to mitigate problems afterward, goes a long way towards making sure you consider all the useful aspects of the problem.

Will you learn any one single lesson as thoroughly or as fast as you would in the real world when it affects you so much? Likely not, but then again, that assumes you actually saw a solution to the problem, and that still might just be one aspect.

There is a cost in not having certain procedures in place. On the other hand the red tape has a cost too.

A single mistake shouldn't lead unconditionally to a mandatory multi-step process that involves multiple people and may take several months to finish. The examples of the "cover your ass" decision making overpowering the common sense engineering are abundant.

It depends on the project. Is it a web application for keeping track of daily tasks? You probably don't need that much rigor. Are you building something that if it fails will harm/kill/cost people a lot of money? More rigor is needed.

That's true, you can definitely over-correct. I guess if I was the supervisor in this case, I'd be doing some analysis with the developer after the fact. Were procedures correctly followed? If not, use this event as an example of why the procedures are in place. If they were followed, was it a brain-dead simple error? If so, then obviously there is something wrong with our test plans. If it was one of those errors that turned out to be a perfect storm of environment differences, unexpected inputs, code side-effects or what-have-you, then there probably is no quick solution and the imposition of some overbearing system to try to limit something that will probably not happen again would be overkill.

It's always best to do an after action analysis to see where things broke down, I think. Changing procedures or adding protections might be warranted, as long as it's not the "get my permission before ever putting a Y in that field again" kind of enhancement.

I feel where you're coming from, but in reality it's not like any test suite can guarantee to prevent downtime caused by bad deploys. All it takes is a little networking related bug, something that differs ever so slightly between environments, whatever, and your site could be down.

I agree with the other replies that while you may be the 'technically right dev' kind of person, it still comes of as being a dick and you will likely not get a positive response from something like this, especially if you caused it.

What makes it more reasonable to say it's the company's responsibility to protect you from fucking up the site instead of you just being a good developer, and not fucking up the site?

It's not about being a good cowboy developer, it's about beijg a professional engineer who builds systems that work.

I don't expect the folks who built my house to detach and reattach the door pefectly every day, I expect them to build a house that works.

Exactly. If you're one mistake away from a disaster, you already lost.

Ideally you would have either test gates, or a staged rollout with automated rollback.