Lessons Learned from Twenty Years of Site Reliability Engineering

> very broadly applicable. I don't see any "this would only apply at Google" in here.

One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

I started getting phone calls at 5AM regarding servers going down. I immediately went to email to connect with other members of the team and found my inbox full of thousands of emails from the monitoring service notifying me of the outage. Those were days:)

We needed to have extra layers of backup communications because the first three layers systems were all also Google properties. That is to say, email wouldn't work because Gmail is Google, our work cell phones wouldn't work because they're Google Fi, and my home's ISP was Google Fiber/Webpass.

All of which is to confirm that, yes, Google has a backup IRC server for communication. I won't say where, but it's explicitly totally off Google infrastructure for that very reason.

One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge.

People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?

They (>1) do exist but not on AWS or any other major cloud provider.

(Or at least that was the case two years ago when I still worked there.)

IIRC some of the information about their approach is considered sensitive so I won't elaborate further.

You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or reset. You end up requiring more automation, version control and change management, which speeds up and simplifies overall work, in addition to preventing and quickly fixing outages. It's a big culture shift.

It's clear that the Internet itself could not be cold started like that AC grids, there's simply too many AS's. (Think about what AS means for a second to understand why a coordinated, rehearsed cold start is not possible.)

A second major reason is automation. If you read the linked site long enough you'll find that in the early days of Google, SREs did plenty of manual work like deployments while manually watching graphs. They were indispensable then simply because even Google didn't have enough automation in their systems! You can read the story of Sisyphus https://www.usenix.org/sites/default/files/conference/protec... to kind of understand how Google's initial failure of adopting standardized automation ensured job security for SREs.

And they likely won’t be as good as dedicated SRE teams.

But few businesses care about that right now considering layoffs.

Throwing developers at a problem even if it isn’t in their skill set is an industry trend that won’t go away and be more pronounced during downturns.

Full stack developers are a great example of rolling two roles together without twice the pay.

I'm a SWE SRE. I think in some cases it is better to be folded into a team. In other cases, less so.

One SRE team can support many different dev teams, and often the dev teams are not at all focusing time on the very complicated infra/distributed systems aspect of their job, it's just not something that they worry about day to day.

So it makes sense to have an 'infra' that operates at a different granularity than specialized dev teams.

That may or may not need to be called SRE, or maybe it's an SRE SWE team, or maybe you just call it 'infrastructure' but at a certain scale you have more cross cutting concerns across teams where it's cheaper to split things out that way.

Still the same competency. Someone needs to know how those protocols work, tell you latency characteristics of storage, and read those core dumps.

Is sre a bubble thing.

I never got why SRE existed.(SRE has been my title...) The job responsibilities, care about monitoring, logging, performance, metrics of applications are all things a qualified developer should be doing. Offloading caring about operating the software someone writes to someone else just seems illogical to me. Put the swes on call. If swes think the best way to do something is manual, have them do it them selves, then fire them for being terrible engineers. All these tedious interviews and a SWE doesn't know how the computer they are programing works? Its insane. All that schooling and things like how does the OS work, which is part of an undergrad curriculum, gets offloaded to a career and title mostly made up of self taught sysadmin people? Every good swe Ive known, knew how the os, computer, network works.

> if SRE will be around 10 years from now,

Other tasks that SRE typically does now, generalized automation, provide dev tools and improve dev experience, is being moved to "platform" and teams with those names. I expect it to change significantly.

Who told you that?

QA isn't going anywhere... someone is doing testing, and that someone is a tester. They can be an s/w engineer by training, but as long as they are testing they are a tester.

With sysadmins, there are fashion waves, where they keep being called different names like DevOps or SRE. I've not heard of such a thing with testing.

I think it’s simply swapping one set of trade offs for another. With dedicated SREs you have true specialists in production operations and their accompanying systems (tooling, alerting, etc) with a clear mandate and ownership of outcomes; but they don’t necessarily have full ownership of what they’re keeping running, and that can cause organizational problems (we want to launch X, SRE says no, or vice versa) and make it so non-SREs take no ownership over their hard-to-support code.

Conversely you can have Eng teams without SREs and most of those organizational/social problems, at the cost of production reliability being only one of many priorities.

I think what’s really happening is that a lot of companies are deciding they don’t care about reliability very much as a business outcome, especially when it comes at the expense (at least in opportunity cost) of less features.

I think it’s definitely one of the aspects.

Talking with some SRE friends the point that they think part of their role is important are the multitude of moving parts in the current development environment (partially related with the easy money for resume driven development and a lot of tech stack side quests) and how the bar had lowered to hire folks (for this one with hiring managers with almost infinite budget e a lot of questionable product initiatives).

... But once your firm starts making guarantees like "five nines uptime," there will be some complexity necessary to devise a system that can continue to be developed and improved while maintaining those guarantees.

At google we also had to routinely do “backend drains” of particular clusters when we deemed them unhealthy and they had a system to do that quickly at the api/lb layer. At other places I’ve also seen that done with application level flags so you’d do kubectl edit which is obviously less than ideal but worked

1. Keep it simple. No elaborate logic. No complex data stores. Just a simple checking of the flag.

2. Do it as close to the source as possible, but have limited trust in your clients - you may have old versions, things not propagating, bugs, etc. So best to have option to degrade both in the client and on the server. If you can do only one, so the server side.

3. Real world test! And test often. Don’t trust test environment. Test on real world traffic. Do periodic tests at small scale (like 0.1% of traffic) but also do more full scale tests on a schedule. If you didn’t test it, it won’t work when you need it. If it worked a year ago, it will likely not work now. If it’s not tested, it’ll likely cause more damage than it’ll solve.

I've done this with djb's CDB (constant database). But I've seen people poll an API for JSON config files or dbm/gdbm/Berkeleydb/leveldb.

This can extend to other big red buttons. It's not that elegant but I've had numerous services that checked for the presence of a file to serve health checks. So pulling a node out of load balancer rotation was as easy as creating a file.

The idea is that then when there's a datab outage the system defaults to serving the last known good config.

To make up an example that doesn't depend on any of those things: imagine that I've added a new feature to Hacker News that allows users to display profile pictures next to their comments. Of course, we have built everything around microservices, so this is implemented by the frontend page generator making a call to the profile service, which does a lookup and responds with the image location. As part of the launch plan, I document the "big red button" procedure to follow if my new component is overloading the profile service or image repository: run this command to rate-limit my service's outgoing requests at the network layer (probably to 0 in an emergency). It will fail its lookups and the page generator is designed to gracefully degrade by continuing to render the comment text, sans profile photo.

(Before anyone hits send on that "what a stupid way to do X" reply, please note that this is not an actual design doc, I'm not giving advice on how to build anything, it's just a crayon drawing to illustrate a point)

I do think Google should be more open about their internal outages. This one in particular was very famous internally.

Maybe things have improved since couple years ago, though

Error budgets also only matter if you either give a shit about your guys or have to pay them for that oncall time. Plenty of employers are happy so say 'salary is exempt, suck it up lol' so errors are effectively free.

https://data.iana.org/TLD/tlds-alpha-by-domain.txt

• .android

• .cal

• .chrome

• .gbiz

• .gle

• .gmail

• .goog

• .google

• .play

• .prod

• .youtube

Google also owns 22 generic domains:

• .app

• .boo

• .channel

• .dad

• .day

• .dev

• .eat

• .esq

• .fly

• .foo

• .hangout

• .here

• .how

• .ing

• .meme

• .mov

• .new

• .nexus

• .page

• .prof

• .search

• .zip