Hacker News Reader

Comment by x87678r | Hacker News Reader

I'm curious about big outages like this in big internet corps.

Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved?

I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.

View on HN

I assume this is in the SRE book, but a tier one product like the identity service will have global SRE coverage (i.e. at least three SRE teams so that there is always an SRE group for whom it is daytime holding the pager). Devs are often involved in diagnosis, but are less often required for mitigation, as the mitigation is almost always to revert whatever change caused the problem. This is a simplification of course, but it gives an idea of the general pattern.

I won't comment on the incident, but I can tell you that we have two, not three, SRE sibling teams each. That still gives awake-time coverage, but not working-hours coverage. We simply pay folks for the time spent oncall outside of working hours. (Google SRE)

4am is noon Europe time, so the sres in Europe would have gotten the page and been on top of their game. They fixed it pretty quick.

Of course in a global outage, nothing is fast enough.

They found the root cause within 20 minutes. I doubt one of the original developers could have been involved that quickly.

Yes, SRE teams typically have a sister SRE team in another continent and time zone.

Reply on news.ycombinator.com