Demystifying Site Outages | Hacker News Reader

Demystifying Site Outages

etsy.com

30 pointsmcfunley11 comments

This is the most detailed server outage response I have ever seen. Looks like Etsy understand the importance of accountability and I really appreciate the way they wrote the post. A short version for non-technical and short attention span minded people and a longer version for the nerds who like to know the intricacies of big sites like Etsy.

And even the longer version was fairly understandable for non-technical folks, I'd have thought. Very well written.

I always remember reading in a book once (Influence by Cialdini?) that the main signifier for whether a doctor gets sued or not is not whether they make mistakes, but how they handle them. 'fessing up honestly is the way to go.

Good on them for taking the time to write such a detailed report. Shows respect for their users.

Without diving too far into technical details, suffice it to say that in order to support the special characters that some languages require to be complete, we needed to do an upgrade of the database server software.

Oh, hi, I see you're using MySQL. MySQL's 1024 character (oh sorry, not characters, bytes, oopsie!) limit for indices was the bane of my existence. You have a mountain of code and tables using varchar(200) and an index spanning five such columns. Works with latin-1. Then somebody decides this new fangled utf-8 is the new hotness. hahahaha, no database index for you! Now you get to go back and decide if you can shrink said columns or get by with a one column index (mysql assumes 3 bytes, so one 200 char column is already more than half your allotment).

The hilarious part is this is just Catastrophic MySQL Production Error Case #693, but people won't dare move away from MySQL. They will keep using it and building out new services based on it because there's such a large list of known failure cases. People feel safe knowing how things can fail rather than seeing something "just work" and not knowing why.

Why do you need to do a MySQL upgrade you say?

"Up until MySQL 5.5 (released December of 2010) the UTF-8 support was severely hobbled." [missing some Japanese characters, including those used in many people's names, and accents]

http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/

On a different but related note, I found that sentence to be pretty ... strange. They go into lots of technical detail about servers, indices, users and tables and things, but that wording sounds as if they're explaining the world to a (small) child.

The idea that there are "normal" (=US ASCII, I presume?) and "special" (=the rest) characters used in human languages is a bit 1980:s, or thereabouts, to me.

I'm guessing the primary audience is the site's users - i.e. people macrame-ing plant holders and such like - and not other techies.

that was interesting, but it felt like two different articles rolled into one (a short non-technical and a long-but-shorter-than-that technical one).

anyway, so alter table modify on a primary key (int to bigint) requires that the index be rebuilt? is that true for all databases? or was the problem something else?

Since the allocation space is different, it's generally a rebuild. The only exception would be a DB storing integers as a variable length int - unusual.

In 2004, when etsy.com was created, we all were pretty aware of the importance of specifying utf-8 when we setup systems.

How could a company, that now has 120+ engineers listed on the about page, get to 80+ databases without fixing this issue?

Reply on news.ycombinator.com