Scary Outage Stories from CTOs | Hacker News Reader

Scary Outage Stories from CTOs

thenewstack.io

74 pointsmooreds13 comments

Oh, here's my worst outage story:

https://github.com/stickfigure/blog/wiki/Beware-cutesy-two-l...

I will never, ever again use a non-.com domain for a business.

Shit. I use a somewhat critical .st domain. I should look into moving off it.

We used to use .so domain actively for our business, but fortunately we acquired .com and moved. We still utilized the .so domain for some internal tools.

One day, out of the blue, the authority started refusing any requests towards the domain. We tried to contacting them, but nothing. It took a week for the domain to come back. If this was our main domain, I think we would be not in business anymore.

The "metrics are normal but users complain" example is something I've experienced repeatedly as a user. It's incredibly frustrating and leaves a really bad impression when an app/site is utterly, obviously, reproducably broken and stays in that state for a week.

The affected users don't care that it works for others, and you can only afford to ignore it if you are OK with not only losing those users forever, but also the negative reviews and word of mouth.

The probably biggest (in terms of company size/affected users) example of this would be Twitter's "something went wrong" on mobile. I'm sure that they have some sort of metrics showing them that everything is fine, but plenty of users on HN complain that they are pretty consistently getting a generic error message when they open a Twitter link until they reload the whole page. (I assume it's separate from the issue mentioned in the article, since it was still ongoing recently.)

Also, because errors are much more noticeable than everything working, a 10% error rate is pretty much perceived as "never works".

A subtopic of that: The affected users don't care why it is broken, and it is broken until it is fixed for them. If you fix the issue, but the fixed version hasn't yet made it to your users' phones (e.g. because it's only fixed in the development version, or your version hasn't made it through some review process), then your app is still broken. (Looking at you, Newpipe and Firefox: Newpipe was broken for weeks, apparently because F-Droid, their official distribution channel, is shipping an outdated version, and when trying to fix that by downloading the newest APK, I ran into a bug in Firefox Mobile that prevents downloads from working - which is fixed in some developer version but not the public one, and has been broken for at least a week).

I once read, a huge problem of data driven approaches is that people tend to measure what is easy to measure and not what should be measured.

One of my own stories: in which I discover that you should never ever attach a clone of a bcache backing device to a live server: https://blog.bitmex.com/postmortem-downtime-july-5-2017/

One of the first companies I worked with had a gigantic, replicated filesystem across multiple Solaris boxes, in multiple geographies. It was actually faster to replicate things with rsync across geographies than do anything else, simply because of how slow compilation could be (12 hours on a tricked out e450).

A simple "rm -rf" turfed a directory full of configuration files. This was immediately (well, 2004 immediately) replicated to several countries. Thus several telephony systems went poof. Obviously the same rm -rf took out the CVS repository, and on a Saturday night. Want to make it more fun - the ENTIRE (worldwide) development team was offsite for a team-building week, in Bonn.

Two of my friends got on a plane, flew to Ottawa, and started having a go at computers to find a recent repository. Hard drives were ripped out of machines, jacked into others, mounted, and prayers made. This wouldn't have had to happen had the tape robots (yup, plural), with backups not both their arms stuck, and the vendor simply not providing 24x7x365 support because the company was too small. On the bright side, everything was up relatively quickly all things considered.

Way better than the time Mike BBQed 20000 chickens accidentally.

You mean a large multinational company in telephony that had a campus in Ottawa?

Nope, we were too small - instead we had folks, offices, and partners all over. We were neither of the two you're thinking of.

Can you tell us about the chickens too Please :3

I find random errors that are hard to replicate in a consistent fashion the most difficult bugs to track down. A while back, a wordpress site I manage would respond with cipher text for certain pages randomly. After several moments of frantic efforts to fix this, I noticed the issue was that a plugin was doing caching, gzip compression while wordpress was trying to serve pages on https.

Horror story here our servers are in a basement datacenter under another datacenter. An intervention on the upper datacenter caused water to flow on our servers on 23th of december.

Needless to say that was a pretty rough week.

I think I was bit by that last one at mobile web Twitter. I read somewhere that clearing cookies helped and it fixed my problem. It's good to know the underlying issue has been solved, however.

Reply on news.ycombinator.com