Hacker News Reader

Hacker News

Comment by tithe | Hacker News Reader

Full thread

tithe

I imagine a conversation with this individual as team lead would go something like this:

"So, you worked overtime to save systems across the planet from crashing due to a botched update?"

"Yes, sir. We're 'Site Reliability Engineering', after all."

"And people in airports don't have to sleep on the floors because airlines can actually schedule flights?"

"Yes, sir. Site Reliability Engineering, at its finest, sir!"

"No, you played the hero. That's bad for the team and normally for you, really. You should have let it break."

"But our team...is 'Site Reliability Engineering'?"

"You should have let it break."

"But, Site...Reliability?"

"You're fired."

View on HN

Being the hero once is fine -- but if the only reason we don't have outages on every update is because of an SRE who takes it upon themselves to babysit the update, then the system is super broken and there will come a time when that babysitting doesn't work and the lack of awareness of the problem will cause the cascade to be that much worse.

I do agree. If persistent heroic acts are a requirement to keep any system running, then let's allocate resources to determine why that's the case and how the system can be changed into something more resilient.

The problem is that it sounds good to talk about "allocating resources" and "communication" and making the system more resiliant. But even identifying places where heroism is being applied is extremely difficult except for the hero. Most importantly, the people who need to know that herois is keeping things alive will often be completely unaware that there is even a problem.

In the end the only way to do this is to allow the system to fail. If you find yourself in a position of being a hero, you have to notice it and do something about it. You could do a big writeup of how to fix the system to remove the requirement for heroism, etc., but as an SRE you don't always have insight into how important the particular issue is, so you could be wasting a bunch of time on something completely unnecessary (or that is not worth your time or the time to fix it).

> In the end the only way to do this is to allow the system to fail.

This may be the efficient way for systems under test, but for a live, production system there must be higher bar of performance than "let it fail". I agree with several of the points Malmberg makes (which my original sarcastic comment probably doesn't suggest), but his final conclusion of "let the system break" is alarming and dangerous.

> If you find yourself in a position of being a hero, you have to notice it and do something about it.

If I found myself being the hero, I would absolutely push this forward and do something about it. It's also a tragedy that this may actually result in the opposite outcome that you want (like being fired for "not being a team player"). At the end of the day, it's still human beings in charge of these sytems, which means handling our communications with grace and tact.

Just from experience, the system failures tend not to be of this mode. If heroism is routinely deployed, then yes, failures can be huge and even more heroism then needs to be applied.

But normally, what failures look like is a degredation in responsiveness or a failure to scale up quickly enough for surging demand or faster turnaround on canary failures or caches that need to be purged after batch jobs, etc.

Much more "degredation below SLA" rather than "every windows machine in the world blue screens". Heroism for disasters like that, sure, but that's going to be a post-mortem and a big deal. Most of the time failures are small, and letting them fail means that generally there is more awareness of the problem -- clearing caches or restarting the instances because they get slow conceals the problem and become part of the background routine.

The note on heroism is not a note for managers -- it's a note for SREs to actively notice when they are engaging in heroism and to stop doing that. Letting the cache get overloaded so that an automated system can do the purges because the development team is now aware of the issue is far preferable. And sometimes these routine acts of heroism become routine process/superstition to fix problems that no longer exist, or that are minor and not worth the time spent.

I also concur. A firefighter putting out a dangerous kitchen fire is heroic, but putting out the same fire several times without finding out the cause is negligent.

More likely: "You worked overtime to save systems across the planet from crashing due to a potentially botched update?"

"Yes, sir. We're 'Site Reliability Engineering', after all."

"And nothing has changed. Updates are a smooth as usual."

"Yes, sir. Site Reliability Engineering, at its finest, sir!"

"You're fired."

Reply on news.ycombinator.com