Hacker News Reader

Comment by Someone1234 | Hacker News Reader

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed?

I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they could likely use the same wage-hours to just hire someone else for a shift. Heck with an international company like Google they could have UK-based staff monitoring US-based sites overnight and visa-versa. Keep everyone on 9-5 and still get 24 hour engineers at their desks.

View on HN

Google does spread oncall rotations across multiple timezones. Most SREs are oncall only during the day, with the local nightshift being somebody else's dayshift.

For a more detailed look at Google's SRE operations, watch Ben Traynor's excellent talk "Keys to SRE": https://www.usenix.org/conference/srecon14/technical-session...

That was an insightful talk, thanks.

As someone who is on a regular rotation - being oncall sucks. But there are definite advantages.

Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engineering calls.

The team can bond over operations - it sucks being oncall, everyone knows that so everyone tries to make it less sucky, by clearing ops queues before handing off or making ticket messages a bit nicer.

When you start having ticket-free weeks or months it is an awesome feeling, the service works and is robust and your team can spend that time writing new stuff.

Additionally, when you have a small team building/maintaining a service it makes far more sense to rotate the oncall responsibility between them rather than an external engineer.

> Having the team feel direct pain is a great motivator for building robust applications

But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues.

You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

There are definitely corporations where engineers who build the applications take responsibility for maintaining and running the applications - I happen to work for one.

This should be all corporations.

> You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

why do you think that should be the case? I'd argue quite the opposite having done this in a large corporation with pretty decent success.

to be successful, it requires that your monitoring isn't noisy; most issues are repaired in an automated way; then when you get a "page" it is something that someone deeply familiar with the code / service is the best person to react to it. it will drive down the MTTR. and it typically gets the root cause fixed sooner in the code. these folks need to be engineers, but they don't all need to be just the devs, since internet scale services just are noisy and you need to have some randomization buffer. but those engineers need to be dedicated to the service and not a central org, where they aren't going to know how to look at logs, have depth on the intra-service interactions, the latest changes, etc. to me, this is also the best definition of "devops".

SRE has veto power over many of the design choices that go into the actual application.

The only people who can support or fix an application is often the team developing the application itself.

Yeah, when I interviewe at Google, this was pretty much what I was told - one of the prices of writing "new, cool stuff" is that you get to support it, because no one else is going to be able to fix it when things go wrong.

Sounds as if that requires retiring systems when the developer leaves the company.

I would imagine that if the new thing makes the transition to being a successful, widely used service then putting a support team in place is a part of that. At the beginning though, when no-one knows whether your new thing is going to be a success or not, the dev team is it.

I think it's simple: Developers who are on-call write better code.

As soon as you take the view of "This is crappy, but keeping it running is someone else's problem" then everything suffers (product quality, engineering quality, and reliability).

There's a lot to be said for having a domain expert be on call. Most issues can be fixed really quickly, but if it's your first time ever seeing an alert or working on a service, you're pretty much boned and end up paging the domain expert anyway.

That's exactly what Google does except some teams have triple rotations like Mountain View/Sydney/Zurich. I'm not sure why you think the word "rotation" means something else.

I've worked for companies in the past where a "rotation" was you were on call 24/7 for a week or longer. Worked all night? Still expected at the office no later than 9 a.m.

It was exhausting as the people on call didn't have the power to actually fix the system. Instead, they would have to walk someone else through the steps over the phone. If it took a code change to fix the system, too bad that was at least two weeks of red tape, and every single time the error occurred they had to page the person on call to walk the person through over the phone to verify it was the same error and nothing could be done.

Think "big business," "division of responsibilities," "accountability," inept management who never had to suffer under the policies they demanded, and a toxic culture which was proud to "give everything they have for the product."

Luckily, I don't work for those shitty companies anymore.

IMO the difference is ownership. That you own your code from the inception through production and that there is never a hand-off. It leads to better code and better moral.

What makes you think this isn't the case? I don't see anything to the contrary in the document.

You cannot hire top-quality engineers who are willing to do shift work.

Reply on news.ycombinator.com