Hacker News Reader

Comment by tialaramex | Hacker News Reader

For me at least 24/7 incident response is completely acceptable in a properly compensated role so long as it's accompanied by the culture that says preventing such incidents in the first place is Job #1

That is, I'm OK with being woken at 0200 to try to understand and if appropriate fix or recover from a disaster only so long as if I'd suspected this might happen the people expecting me to be awake at 0200 would have given me the resource (money, people, whatever) to fix it. If I feel like I don't have that support, I'll only start looking at your disaster during my working day.

My impression is that ISRG pays a lot of attention to preventing disasters, so if I worked for ISRG (not very practical since they're based on the US West Coast and I live in England) I'd be comfortable taking a call in the middle of the night to fix things.

View on HN

operations based on US west coast could definitely benefit from a few people on the other side of the world to achieve a 24/7 coverage while keeping a good work-life balance.

Local nerds can be noctournal too. Letting people pick their preferred shift is just as important as accommodating other kinds of physiological diversity.

Can confirm.

You'd normally want the 24/7 people to be part of the day to day operations, otherwise they will quickly stop being up to date, so their don't go out of the knowledge loop, so selecting the reasonable timezone set is not trivial.

yeah, my assumption was that the team in another TZ would also work together on the same thing. Yes it has some challenges, but there are a few upsides beside the oncall coverage (e.g. increase in talent pool)

Yeah, as long as it doesn't happen often. I'm technically always on call but we haven't had an on call incident in close to a year.

Basically I keep a phone and laptop on me at all times.

This is in comparison to a friend that works somewhere that always has daily on call incidents that are not actually problems 95% of the time. That would piss me off even if I weren't always on call.

It's not my job to decide what counts as a "problem", but if the boss wants me to handle the CTO drunkenly leaving her phone in a taxi as an "emergency" to react to at 0200 then as described above I want the resources to handle that in advance when I predict it'll obviously keep happening. If my boss doesn't want to divert people or money on preventing such bullshit problems, then I don't want to get woken when they happen.

At huge scale there just will be incidents every day, if you're SRE for Google then every day is a bunch of problems - but at that scale those are routine incidents and you can afford the follow-the-sun team (actually I think Google SRE found it was better not to have true follow-the-sun but just two groups) to handle them, leaving only still rare "San Francisco just fell into the Bay" level incidents to actually wake people up.

Reply on news.ycombinator.com