Airbnb Tech Talk: Jeremy Edberg of reddit and Netflix -- Building with Clouds | Hacker News Reader

Airbnb Tech Talk: Jeremy Edberg of reddit and Netflix -- Building with Clouds

airbnb.com

54 pointsjedberg35 comments

I know I referred to myself in the third person, but it was easier than expecting people to look at the username. :)

Also, if you're attending or watching at home and there is something specific you're hoping I will cover, tell me here in the comments and I'll do my best to cover it!

Just remember, 75% of the folks here on hacker news won't be attending your event, so following up with the slides/video would be much appreciated and would go a long way of increasing your reach!

If 25% of hacker news folks attend, then the talk will have about 100,000 people. Jeremy - you better get a whole football field for your talk!

If you're ever in the area make sure you attend! Airbnb puts on a great evening... beer, food, and tech BS'ing. Doesn't get any better!

Honestly, I think your username is more recognizable than your name. I "know" who jedberg is but until I saw this post I didn't know your name was Jeremy.

It's funny, I was going to put my username in the title, but that just seemed a little too ... much.

Did Reddit overcome its rather serious persistence-side issues?

It seemed like the object-blob Postgres store was a disaster, then they brought in Cassandra which was initially used as a cache of sorts, but now is the primary point of contact for threads apparently?

Reverse-engineering the persistence strategy from the open sourced code is more tedious and potentially inaccurate than I am comfortable with, so could a Reddit employee or code contributor provide some insight as to how those problems were worked past?

There was Reddit downtime today (read-only mode), which is common. What's the usual cause of that sort of thing these days?

I can cover the rest in the talk, but I just want to comment that the Postgres object-blob store (as you describe it :) ) was not only NOT a disaster, but one of the strongest parts of the persistence layer.

Then where did the growth pains that necessitated Cassandra come from? I'm trying to build up a mental genealogy of the engineering decisions made so that the knowledge isn't lost to future programmers. Most of you OG peeps have moved onto greener pastures.

I'll cover that in the talk, since I'm sure others would like to hear the answer as well. Thanks for the suggestion! If I don't cover it sufficiently, you can hit me up for more info.

> Then where did the growth pains that necessitated Cassandra come from?

Increased traffic? I don't think I understand the question

Mostly porn and ragecomics, wasn't it?

And advice animals

A little less porn these days, no?

>Increased traffic? I don't think I understand the question

I was trying to determine what component broke first.

There's always some component "breaking". That's just how performance works, you're always bounded by some resource.

An app like reddit has most of its backend tweaked or rewritten continuously as it becomes more important to find and alleviate some new bottleneck. If you do it right, you predict the next bottleneck before it becomes one but there will always be something.

Maybe writes take too long. Why? You've capped the performance of the disc in your DB master (or more importantly of the most expensive single set of discs you can afford). Maybe adding app servers proportional to site-wide traffic stopped helping you keep up at some point. Turns out you're network bandwidth bound. Maybe requests are too slow, but apps aren't using much CPU. Why? Because you're network-latency bound. Hey, it turns out that some tight loop was doing single memache GETs when it could have been doing a single multi get.

That last one is just a general performance bug, but that's just it. Every bottleneck that keeps you from just adding resources is a performance bug. Assuming your app is not infinitely fast, there's always something.

ketralnis will likely remember this better, but by my recollection we switched to cassandra after we hit the limit of what we could do with memcachedb:

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-f...

We were already using memcachedb as a persistent keystore, "caching" precomputed listings and comment threads. When it started becoming more trouble than it was worth to maintain, we decided to try cassandra as a beefier replacement.

It was capping memcachedb's performance that was the last straw, but it became pretty clear that we were going to similarly cap the performance of any disc that we could afford to put in a postgres write master and that we needed a solution that eliminated that bottleneck, as well as it as a single point of failure. Cassandra (or more generally the dynamo model) did that nicely.

As I remember it, most of the downtime and hairpulling from mid-2009 to mid-2011 was due to EBS issues, wasn't it?

Now, maybe that was Amazon's fault for selling a flaky storage solution, or maybe it was our fault for not getting our databases the hell off of it, but yeah, it definitely wasn't Postgres's fault.

I'm gonna go page ketralnis, since he got his hands a lot dirtier than I did with respect to reddit's storage technologies.

A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS.

As a side note, things have been much better, from what I hear, since they moved to local storage. Things work much better here at Netflix for the same reason.

I haven't had much of a chance to play with the new Provisioned IOPS much, but from what I've seen that is a pretty solid storage solution.

> A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS.

This bug was a pretty rare case, but to be specific, here it is:

1. Databases for a particular data type are specified in the config file like:

  dbmaster, dbslave1, dbslave2, dbslave3

2. In Python this ends up just like that: a list where the 0th element is special. There's a "config list" (the list read out of the config file) and an "active list", which is those from the config file that have recently reported themselves as live.

3. Reads are sent to (sort of) random.choice(active_list) and writes are send to active_list[0]

4. But wait! If the master is "down" (the EBS backing him has slowed down so far as to mark him unresponsive) then active_list[0] may very well be a slave

5. Londiste now can't reconcile the slave database with the master database. There's a sequence conflict!

The symptom would be that a replica would refuse to replicate, we'd detect the replication lag on that slave, and remove it from the active list. Then its load would fall, we'd get an alert, and we'd say "well crap" and rereplicate to it.

In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems.

> In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems.

It got progressively worse as time went on the EBS performance got worse. It was happening every couple of weeks after you left, and I think it got even worse after that.

Luckily we had dug into the londiste internals and managed to figure out how to restore replication much faster, but that was still super painful. :)

But you're right, the disk lag usually caused the immediate problem -- the out of sync slave was usually pretty easy to deal with, depending on how many got out of sync at once. It was just annoying and time consuming.

As an aside brought on by the fraternal ballyhoo going on here, do the ex-reddit guys still keep in touch? It's kind of sad that Eric (and, I suppose, Alexis) are the only members of the old guard still kicking around there.

> do the ex-reddit guys still keep in touch?

Many of us are at Hipmunk now, so at least that many. We're also still friends with each other and with the current reddit folk

We all see each other from time to time, often at events thrown by the current reddit folks.

> Did Reddit overcome its rather serious persistence-side issues?

Mostly by getting off of EBS

> It seemed like the object-blob Postgres store was a disaster

Goodness no

> then they brought in Cassandra which was initially used as a cache of sorts, but now is the primary point of contact for threads apparently?

I don't know what "primary contact for threads" means, Cassandra doesn't really do IPC.

Some datatypes are in Postgres (Links, Accounts) and some datatypes are in Cassandra (Votes, Saves/Hides).

> There was Reddit downtime today (read-only mode), which is common. What's the usual cause of that sort of thing these days?

I can't speak for today now that EBS is mostly out of the loop, but it used to be EBS issues on a DB master

Dang, all sold out. Got on the wait list, hopefully I'll be able to snag a spot.

Since Reddit has been down a lot today... awkward.

I've had nothing to do with reddit's operations for about 16 months now. I didn't even know it was down today. :)

Also, it's been down way less since you left. Coincidence, or vastly increased budget and staffing? We give you the facts, you decide at 6.

Have I wronged you in the past? I ask because pretty much every time there is a thread about reddit or Netflix tech that I comment on, there you are slinging insults at me (which are incorrect, BTW).

I'd love to know what I did to harm you so that I can apologize.

Or do you just like giving me a hard time for the lulz?

I've noted redthrowaway to be oddly but harmlessly enraptured by morsels of Reddit inside baseball[1]. You may be mistaking affectionate ribbing for spite.

Or maybe I'm missing someone's dickishness, who knows.

1. An exemplary specimen from this very thread: http://news.ycombinator.com/item?id=4681972

That's pretty much the gist of it, yup.

That wasn't intended as an insult in the slightest, and I apologize if it came off as such. It was meant to be a light-hearted joke. I also don't recall any other threads where I may have given you that impression, but apologize (again) if I have.

To clarify, I'm a long-time redditor who was pretty impressed with how well you guys managed given the small team you had.

Ah, alright then. Looks like danilocampos was right.

My mistake! :)

Reply on news.ycombinator.com