Hacker News Reader

Comment by jedberg | Hacker News Reader

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here.

> COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!!

Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one we used was unavailable.

And I think Google has a backup IRC server on AWS, but that might just apocryphal.

Always good to make sure you have a backup side channel that has as little to do with your infrastructure as possible.

View on HN

God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.

Imagine if they accidentally deprecated their last messaging service? Can't let that happen so better build some extra just in case.

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my team's system is down, it's not improbable that Google Meet (along with Google.com) was down because of it.

We needed to have extra layers of backup communications because the first three layers systems were all also Google properties. That is to say, email wouldn't work because Gmail is Google, our work cell phones wouldn't work because they're Google Fi, and my home's ISP was Google Fiber/Webpass.

All of which is to confirm that, yes, Google has a backup IRC server for communication. I won't say where, but it's explicitly totally off Google infrastructure for that very reason.

Hurricane Electric? He, he, he.

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect.

As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”.

If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.

At my not Google job we talk about "what happens if a meteor hits a DC".

We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that.

However your SRE teams needs a way to recover without intervention which is why there is talk of backups.

BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.

But that’s incredibly stupid. A meteor hitting a DC is an intentionally dumb way to eclipse the much more likely risks of thousands of other things that can wipe out a DC.

I’ve been involved in such discussions and on the surface they seem reasonable but it turns into an easy reason to write of DC being wiped out.

In comparison there are far more likely reasons for a DC to be wiped out effectively permanently. Data centers at the base of WTC 1 and 2 are good examples. The myriad of targeted attacks on the power grid are also prime examples of attacks that would cripple a data center for weeks if they were in the cross chairs.

The Cascadia subduction zone has a much higher probability of wiping out all of them in the PNW simultaneously than a meteor hitting a single one.

Looking at it from a different perspective, there are other teams at Google who understand how sensitive their data centers are and they act appropriately. The list of data center locations is not public. A motivated group with long range rifles purchase from Walmart could wipe out a Google data center.

Google is very bad at modeling long tail events.

Meteor isn't real. It is literally just a metaphor and more fun than "we lost power" or "egress died". Anybody claiming it is impossible is silly.

You also won't lose a DC a year so you need some strict uptime guarantees to justify automatic roll over.

Prepare to avoid dataloss but needing SRE to change over is fine if you can stomach a hour of downtime.

> Meteor isn't real

Yes, just like "bus factor" isn't about buses

> The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent.

Not sure this is an "only at google" thing. In a past life, I ran Engineering for an App/SMS messaging application. We definitely used an IRC channel as well since the nature of our outages would mean messaging channels could be down. It's also why we didn't rely soley on SMS for our on-call alerting.

I think they meant it’s only at Google that Gmail would not be an uncorrelated email etc.

Though I think Microsoft probably has the equivalent problem with Exchange, Teams, etc.

I wouldn't say this is a Google only problem. Really, all of these problems of redundancy and recovery have been best considered by militaries well before private companies started to have sufficiently global operations and communications networks that they started thinking about it, too. In the Army Recon Course, they taught us to bounce signals off the ionosphere using hand-held radios if we needed to communicate with an otherwise cut-off remote unit too far past the earth's curvature horizon to get line of sight. And, of course, every unit is drilled in contingency plans to the point that it can operate independently as necessary even when all communication gets cut off and continue to push the overall mission forward even in the absence of further coordination, which, for what it's worth, obviously does present the potential for problems of its own when the mission changes and a disconnected unit doesn't know it, as well explored in Doctor Strangelove.

Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere.

> very broadly applicable. I don't see any "this would only apply at Google" in here.

One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

> Afair Google just ran irc on their corp network which was completely separate from prod

I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?

I don't think zero-trust prohibits network segmentation for redundancy or due to geographical constraints etc. It's mainly about how you gain access.

Correct. At an old job we did zero trust corp on a different AWS region and account. The admin site was a different zero trust zone in prod region/account and was supposed to eventually become another AWS account in another region (for cost purposes).

I can’t say if any of this was ideal but it did work unobtrusively.

They do. But I'd say most employees go their whole career without needing to do anything that requires a VPN.

It's basically all web based access through what is, at the end of the day, a http proxy.

SREs need to be ready for stuff like "hey, what if the big proxy we all use to access internal resources is down?".

Way back when, for a while, our local (Google) office's internet access ran off the same physical lines as the local prod datacenter traffic. So, any time there was a datacenter traffic outage of any kind, our office was also out. There weren't a lot of outages of that variant, but we knew immediately when one was happening. It's not particularly fun to have all of your access go out concurrently with a prod outage.

When I was there, the main IRC ran in prod. But it was intentionally a low-dependency system, an actual IRC server instead of something ridiculous like gIRC or IRC-over-stubby.

i think it had a corp dns label but i'm fuzzy on that. yes it could've been a prod instance which would mean you'd need to go to panic room in case corp was down but maybe that was the intention.

Before divestiture we had two groups splitting responsibility for IT, so our DevOps team (bleh) only had partial control of things. We were then running in heterogenous mode - one on prem data center and cloud services.

One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge.

People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?

> And I think Google has a backup IRC server on AWS, but that might just apocryphal.

They (>1) do exist but not on AWS or any other major cloud provider.

(Or at least that was the case two years ago when I still worked there.)

Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.

Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users...

Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second.

And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no complex parsing involved - it is literally just "forward this 50 byte message into this list of TCP connections".

If they're all internal/authed users, you can hopefully assume nobody is deliberately DoSing the server with millions of super long messages into busy channels too.

IRC servers are single threaded. You have contention at that point.

Some of the bigger irc channels have thousand users and this was in the day of 128MB of ram.

Sure it's possible. But you need to test... For example there was/is a 1k file descriptor limit per process in a lot of distributions..

Let's not forget IRC channels can be split between servers too - don't wanna complicate the backup system too much but this is in the original design of IRC.

Brings back memories.

I started getting phone calls at 5AM regarding servers going down. I immediately went to email to connect with other members of the team and found my inbox full of thousands of emails from the monitoring service notifying me of the outage. Those were days:)

Yeah, I work for a CDN and we have to make sure our status page provider is not one of our customers.

For small startups, WhatsApp group or phone call can act as a backup.

Reply on news.ycombinator.com