Hacker News Reader

Comment by mystifyingpoi | Hacker News Reader

Unfortunately the real world isn't black and white. Yes, according to the company policies, I should watch the world burn and do nothing, while looking at the company bleeding money due to customers SLA being broken. Of course, after submitting a ticket to get nmap approved, which takes days. Extra points if I'm on oncall, then racking that sweet incident money is great.

But the underlying SRE culture here is that, if you know what you are doing and have a functioning brain of a responsible person, you'd be forgiven a jump over the fence, if it means putting out a fire on the other side of it. We aren't kids.

View on HN

There’s a middle ground. Get the appropriate stakeholders involved in the decision, including security. Let security be the ones to keep the system down, if it cones to that. Or, let the business operations folks make the decision to go over security’s head. Either way, this is not something an engineer tasked with fixing an outage should be making the decision on.

> this is not something an engineer tasked with fixing an outage should be making the decision on

I don’t get this at all.

I’d much prefer a team of highly empowered and highly responsible engineers than impotent engineers who need hand holding in case they make a mistake.

Well, good thing that wasn’t what I suggested.

Engineers _should_ have leeway in how they resolve issues. As I read, though, you have a company policy which explicitly disallows the action you needed to take to fix the problem (if I misread, my apologies). Getting the stakeholders involved is the responsible thing to do when policies need to be broken.

Ideally, the way this kind of situation gets handled should be documented as part of a break-glass policy, so there’s no ambiguity. If that’s not the case, though, the business should get to decide, alongside the policy maker (e.g.: security), whether that policy should be broken as part of an emergency fix, and how to remediate the policy drift after the crisis.

If you’re all tight enough that you’re allowed to make these kinds of decisions in the heat of the moment, that’s great, but it should be agreed upon, and documented, beforehand.

Well I found out the hard way that company culture or values can mean nothing if you don't CYA. Granted, the shop was small enough that our team was in charge of both the security policies and ops, but still, on one unfortunate occasion I've stepped outside my area of responsibility to "do what's right" and got punished. The next time I've been in a similar situation - well, I've walked away from the fire and grabbed the popcorn.

By the way, I'm still burnt out. This work is stressful. Don't let it take away what's already scarce for you.

Reply on news.ycombinator.com