My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

Yeah, I think this actually may explain things better than specificity. Even if the false alarm rate is low, the rate of actual, life-threatening fires is incredibly low in modern day, so chances are almost every time you hear a fire alarm go off it's alerting you essentially erroneously. I get the impression that in a home the "standard procedure" for dealing with fire alarms is to open windows, fan the smoke away from the thing and/or take the batteries out of the alarm.

So all we need to do is have car alarms be sent directly to you (to your phone or your house) and be loud.

If you're in the US, watch out. I think there are laws against tampering with fire alarms.

(Though I think you're in a ridiculous situation)

> Not even pushing the button helps.

If it's like every other smoke detector I've encountered, the bottom is to test the alarm, not to cancel it.

Your lucky, when someone in my building triggers the fire alarm for there flat, the really loud alarm sounds in all the flats in the building, with an audio note saying that someone has triggered the fire alarm and you can leave if you want, but it's not required. I guess it will tell you when they have confirmed it's a real fire, and you really have to leave.

Not necessarily recommended, but wrapping smoke alarms in plastic bags usually prevents them from going off. Of course...

Near a kitchen, avoid ionization-based smoke alarms like the plague. Get photoelectric instead.

Somehow our rented house has three separate smoke detectors clustered around the kitchen, all of which have been renamed cooking detectors, since the slightest hint of heat seems to set them off, triggering every other detector in the house to alarm as well.

They do at least have a snooze button on them, but I was pretty close to buying a set of the Nest alarms out of my own pocket and replacing them until I saw the price tag.

Unless you have some fancy/aftermarket thing, it sounds just like your horn which you probably hear every time you use the key fob to lock it.

Whether you have the ability to differentiate your particular tone of horn from all the others is a whole different story.

I did. And I see the words "A human just has to parse through a few lines of text ...".

Why is a human doing this? Why isn't the tool identifying the cause through a pre-determined dependency tree and only alerting on the salient points instead of relying on a human intelligently parsing through this data when they've been woken up by a page at 3am?

My experience has shown me that this kind of parsing can be avoided by using the right tools.

Perhaps not the best choice of problem to use as an example - a Database server disk is bad problem has one real solution, and will cause a whole flood of seemingly unrelated problems.

Knowing immediately that it's the server disk (more realistically a raid array going into recovery mode) will save you a lot of time and effort troubleshooting what would appear as a sporadic slow response issue. There's dozens of potential causes for poor response times, of which a raid array in recovery mode is just one.

And once you know it's a raid array in recovery mode, you can then take immediate action, something you can't do if you are still busy troubleshooting a sporadic response slowness issue.

I feel that ultimately, there's no problem with monitoring for high level symptoms, but they should not be the goal state of monitoring. The goal should be to monitor all possible causes of problems to limit the troubleshooting the SA has to do at 3am when woken by a page.

Plus, you should be using a tool which properly silences high level symptoms if there's a problem with a system which is clearly identified as a parent. That is to say, a server being down will silence "db is not responding" alerts.

> Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server.

Sorry, I should have been more specific. DB daemon being down is a cause, and should be monitored. Hardware down is a cause, and should be monitored. Network availability is a cause and should be monitored. Power outage... you get my drift.

I think the root cause of my disagreement with this document is the lack of a proper dependency tree in the alerting tool. Their tool appears to want to alert on any and every monitored problem, which necessitates limiting what you monitor for fear of a pager flood. A proper tool can address this problem correctly.

can't upvote enough. not every monitoring check has to page. this is the key insight -- more visibility is always better, more alerting is not.

This works only as long as you have a single application, or a bunch of independent applications, each with their own team. With any kind of "platform" or "framework" or shared service architecture, the incentive will be for the devs on the application teams to do as little logging as possible, because what will inevitably happen is that failures and errors with "unknown" causes will be marked as platform failures and will wake the platform team in the middle of the night. At that point, you're back to where you started. I've been in that scenario multiple times, and believe me, there are few things worse than trying to debug someone else's code at three in the morning to try to determine if the page that woke me up was a legitimate platform issue or if it's due to application code that's misbehaving or misusing the platform.

Making devs carry pagers certainly helps, but it's a mistake to think that it's a panacea, and that forcing devs to carry pagers will suddenly make them write code with perfect logging and perfect error handling.

Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.

Logging is super fucking noisy and generally not structured for operations support. The state of the art at my not-google employers is basically Nagios scripts. Everyone has these scripts that check various components:

- is the webserver running - is it responding on port 443 - does it return HTML' and maybe - 'If I submit a search, do I get a result back?'

Nagios scripts are responsible for everything: opening network connections, querying system internals, collecting metrics, interpreting results, and boiling it down to a number between 0 and 3 and an unstructured text output to stdout.

A few of us understand that what we need is a more structured, data driven approach. Collect base metrics first, build a time series, apply a projection, and feed that projection in to a system that understands the actual failure condition.

As an example, imagine you're monitoring /. Nagios runs NRPE, NRPE looks at df /, and if it's 85 percent full (by default), sends a warning page. At 90 percent it sends a critical page. A smarter system collects the df / results, delivers it to a central timeseries database. The new data point is used to create a new projection, and the new projection is used to determine the time to an actual failure. The system above might have an idea of how long it takes to respond, repair, and resolve and issue a page when the disk will fill up if not responded to within 4 hours. That's the ideal solution, IMO.

It doesn't exist, AFAIK. There's a massive backlog of scripts that were written in the monolothic Nagios model that need to be rewritten, and thus this newer better version is always imaginary.

We use Splunk at my organization to handle alerting and paging (and lots of other things).

Generally speaking, it works out pretty well for us. If you're a Splunk query guru you can also correlate and/or combine multiple disparate logs in elaborate ways to create more complex alert conditions.

The same can presumably be done with Elasticsearch/Logstash/Kibana.

We're actually security incident response, not reliability incident response, so our goals and methods differ a bit but the core concepts are all the same.

If something not-so-bad-this-time happens while I'm eating dinner, I am not taking time right then and there to figure out exactly when we should and shouldn't alert for that event. That demands a level of analytical thinking that can take place during work hours.

I think anything that requires "spotting potential problems" is only a partial solution. I've never seen a compelling system that can look at all the metrics and (with reasonable precision and recall) spot and summarize changes that are actually problematic and surprising to humans. It's definitely a necessary part of observing what's going on (and quickly eliminating hypotheses like "maybe we're out of CPU!"), for sure.

The subcritical alerts I think of are more things like "Well, the database is _getting_ full, but it's not full yet." Or to borrow someone else's example, "we put in this daemon restarter when it was dying once a week; now it's dying every few minutes and we're only surviving because our proxy is masking the problem but soon it's going to take the whole site down."

So he's talking about system testing and not end-to-end testing. I suppose if your application is really simple, system testing is fine. But if your QA group ever starts automating tests, it's time to re-evaluate.

Blackbox monitoring is (imho) only appropriate for 3rd parties. If it's part of your company, it shouldn't be a black box; that means someone got lazy and didn't demand the devs provide an API.

Also, i'm sorry but this really gets to me: at what point are we talking about 'at scale' ? I think it's whenever tons of money is riding on your site's availability and an unexpected failure causes customers to complain. Immediately VPs start screaming "WE NEED TO SCALE UP!!" and then they mandate some half-assed implementation of the comprehensive monitoring solution they claimed was unnecessary just a month before. But maybe i'm just jaded.

Yeah, when I interviewe at Google, this was pretty much what I was told - one of the prices of writing "new, cool stuff" is that you get to support it, because no one else is going to be able to fix it when things go wrong.

> Having the team feel direct pain is a great motivator for building robust applications

But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues.

You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

That was an insightful talk, thanks.

I've worked for companies in the past where a "rotation" was you were on call 24/7 for a week or longer. Worked all night? Still expected at the office no later than 9 a.m.

It was exhausting as the people on call didn't have the power to actually fix the system. Instead, they would have to walk someone else through the steps over the phone. If it took a code change to fix the system, too bad that was at least two weeks of red tape, and every single time the error occurred they had to page the person on call to walk the person through over the phone to verify it was the same error and nothing could be done.

Think "big business," "division of responsibilities," "accountability," inept management who never had to suffer under the policies they demanded, and a toxic culture which was proud to "give everything they have for the product."

Luckily, I don't work for those shitty companies anymore.

Ahhh I thought the medical profession had it figured out, but these unreliable pagers could be why the doctor is always late for the delivery.

This sort of mirrors how we do things at PagerDuty. Phone may be more reliable than SMS, but every telephony/messaging gateway fails sometimes. We use something like a dozen different phone/SMS gateways to prevent single points of failure, and do end-to-end testing of our SMS providers to check their uptime and latency.

Responders can customize their notification methods (push, SMS, phone, and email) and rules, so you can do things like get a lightweight push notification when an alert happens, and then a phone call 2 minutes later if you haven't acknowledged the incident. Teams get escalation timeouts that forward alerts up the chain if the primary hasn't responded after a period of time.