Yes, it was an SRE team. All we do is write tools to make operations better, but more importantly we write tools to make it easier for the dev teams to operate their own systems better. But yes, we had products teams that would push back on our requests because they had product to deliver, and that was fine. We'd either figure out how to do the work for them, or figure out a workaround.
> We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;)
Well yes, that's the idea. You can't get to 5 9s of reliability unless it's all automated. :)
> I'll give you a simple example of where you can't write code to solve this sort of thing.
I could easily write code to solve the thing. Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up and look at the graphs and figure out what the better medium term solution is to get us through until the morning, including links to said relevant graphs.
You're not gonna have a cost overrun doubling the limit for one night. And if there is a big problem, the person will get paged again a few hours later and have more information to make a better decision.
> But the SRE model asks for operations automation to stick within operations teams, not development teams.
Yes, but I'm not sure I see why that's bad. I don't see any purpose for a dedicated operations team, especially a follow the sun team. If you're Google and you already have offices all around the world, sure, it will be better. But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone.