Hacker News Reader

Comment by yellow_lead | Hacker News Reader

Maybe companies should alert on an increased amount of traffic to their status pages.

That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.

fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible

Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers.

Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.

And the best customers for this are the pickiest annoying ones

Generally I agree. Humans are much better at recognizing patterns than machines.

But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them.

So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)

It’s not last resort, you should do both because you’re alerting on different things.

I’ve seen alerting based on the number of Twitter mentions too

I think this is how downdetector works, it just looks for tweets that are complaining about something not working.

I'm fairly confident they just count a site as down based off the popularity of the page.

That's honestly a good idea. Does anyone do that?

A telco I used to work for did this like two decades ago but even better, they mapped incoming support calls (customers) to stations, and if more than N came in during a certain period for the same DSLAM it triggered some kind of alert.

Same thing happens (should happen) when you visit your ISPs website and look for registered downtime — many requests from one zip code from multiple ips should trigger an alarm if the isp is competent.

In my last job our main “is everything okay?” monitoring dashboard had RPS, latency, 500 rate etc and a line for the number of user issues reported per N requests served. We didn’t alert on this but it was a useful way to detect issues that were subtle enough not to trip up our main alerts.

My company does. :)

I'm surprised more folks don't monitor their vendors via uptime checkers - worth knowing if you'll be able to get anything done that day.

Close that loop: Once the traffic goes above a threshold transition it to red!

They should also monitor HN's submission with GitHub mentioned. Judging from historical pattern it is probably faster than monitoring traffic on status page or even twitter.

Reply on news.ycombinator.com