Hacker News Reader

Comment by lovedswain | Hacker News Reader

The global outage drinking game:

bad DNS config push - 1 shot

routing loop - 2 shots

third party advertising your routes - 3 shots

power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots

Wolves ate through fiber - 5 shots

And it was a full moon - 6 shots

Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots

And service developers left the company to found a startup - 8 shots

Expired internal SSL certificate - 9 shots

Daylight savings changeover-induced database corruption - 10 shots

Windows Update - 11 shots

View on HN

5 is often broken down to “rain fade” and “backhoe fade.”

My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”

In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart

I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.

Coincidentally, I'm in the same predicament with Facebook, itself.

I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.

Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.

isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?

Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.

I was going to link a related comment but you're actually the author of it!

Here it is anyway: https://news.ycombinator.com/item?id=22316230

Please, please tell me this is a true story. It's amazing.

I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.

> Wolves ate through fiber - 5 shots

I hope it's wolves, that would create a new category in https://cybersquirrel1.com/

You forgot:

Hunter shot your fiber relay with a hunting rifle - 6 shots

6 shots? It only needed one.

depends on the gun... some people have interesting ideas on hunting.

DST can be such a hassle. It affects our reports and change requests.

An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.

Don't forget cosmic rays!

Apple decided to shut down Facebook.

Reply on news.ycombinator.com