Steve Huffman on Lessons Learned at Reddit | Hacker News Reader

Steve Huffman on Lessons Learned at Reddit

carsonified.com

202 pointsgiu31 comments

Interesting. I'm an indie Facebook developer. My most popular app has over 8 million monthly uniques and 350 million pageviews. I didn't realize how substantial said traffic was until reading this article. I scaled my app to that size on my own over the course of 12 months. I know of other FB devs who scaled bigger apps working solo. If average hackers like us can do it, anyone can. The hard part is building a popular app. Articles about scaling are a means for founders to brag about their success without giving away useful information about what made their app popular in the first place.

Alexis Ohanian has talked about what made Reddit successful many times. Steve Huffman is talking about the technical side of things. It doesn't come off as bragging to me at all.

Bragging is the wrong word, but it is the first that came to mind. And I was excited to brag about the size of my app when I first read his article.

I just think we spend too much time talking about scaling and not enough time discussing virality.

Scalability stories are generally interesting, as they're about solving problems that out-of-the-box commercial software doesn't. They're interesting to hackers, not all of whom are entrepreneurs.

If you'd like to share a story, however, on using statistical techniques to measure and act on virality, doing multi-variate testing, et al that would be very interesting.

Care to blog about your experiences?

Different application have different scalability characteristics (how much can be cached? how much integration is there?). A certain number may look very impressive for one sort of application, but not so impressive for another.

E.g., some of Yahoo's non-search properties receive tremendous traffic but their scalability stories (while still requiring a great amount of difficult engineering work) aren't as interesting as those of Facebook or other Yahoo properties e.g., Search, Flickr.

Wow, it sounds like your app is extremely popular - that sounds like huge traffic, especially for "scaling to that size in 12 months". Right? I am naive, or is this "fairly common" with decent Facebook games?

What is your application? Are you profitable? Very profitable? (Oh god I hope you are very profitable, or some of my apps are screwed - but then again I guess it really depends on the business model and usage characteristics)

The approach he recommends - no scheme, extreme denormalization, lots of redundancy - is a strong case for nosql.

Reddit in fact now runs on Cassandra - the top nosql solution for a social website

That's interesting. The guys at reddit previously compared various database systems including Amazon's SimpleDB, and found that PostgreSQL was the fastest (see 'Scaling your Python application on EC2' from PyCon 2010: http://pycon.blip.tv/file/3257303/ )

Appearently Reddit switched over to Cassandra in March 2010 (http://blog.reddit.com/2010/03/she-who-entangles-men.html)

To clarify, Reddit replaced memcachedb w/ cassandra entirely, but a lot of the site is still on postgresql. My understanding is they plan to move more to Cassnadra gradually.

Shouldn't lots of redundancy be a fundamental part of which ever db you are using.

As to extreme do-normalization, colour me sceptical, why not parallelise and throw more hardware at the problem?

Because it's more cost effective to de-normalize then to parallelise in this instance.

Turns out it's difficult to parallelize without also denormalizing...

*schema

a note for Mixergy: this is how you provide a video transcript.

Checking it out now.

Thanks Mark.

Their strong usage of both Memcached and Memcachedb makes me think of my favorite 'all-in-one' solution: Redis.

Cassandra (which they moved to in the end) is also an excellent all-in-one solution. It even supports Hadoop since 0.6.

Let's see:

- crash and let a supervisor restart you

- isolate components via specialist processes and systems

- keep schema/types flexible

- be stateless

- cache & denormalize

- be redundant

- let other processes handle non-realtime tasks in the background

Sounds like he spent a long time learning the hard way that reddit should have been written in Erlang in the first place :)

Well, they use (or used to use) RabbitMQ, which is written in Erlang.

So many lectures on scaling are full of best-practices that are somewhat removed from reality and time/budget constraints. This lecture on the other hand has a lot of common-sense advice and solutions that may be obvious to some, but I must admit, this is going to help me (if I could get the traffic to worry about scaling...).

Direct link to an iPhone watchable video: http://vimeo.com/channels/carsonifiedtv#10506751

Joe Stump gave a similar talk at EuroDjangoCon last year (the topic is officially: dealing with a fuckload of data, but a lot of the same lessons). Unfortunately I have no idea what became of the recordings.

There’s also Scaling your Python application on EC2 from PyCon 2010: http://pycon.blip.tv/file/3257303/

Did they say what forced them into emergency read-only mode this week?

He says (transcript is below the video) that he left Reddit in the Fall of 2007 but continued working as a contractor until the Fall of 2009 so he doesn't really address the current events of Reddit.

That transcript is misleading. Steve stayed with reddit until Fall of 2009 when our contracts ended. He didn't leave to work as a contractor.

One thing I should also mention is that our deal was structured in such a way that either one of us could've left reddit before our contract ended without too much pain (missing some carrots, granted) but we were still rather attached to reddit, the community, and our friends (team reddit). Personally, I did also feel a compulsion to fulfill my side of the contract and make sure Condé Nast got its money's worth. That may make me a tool in the eyes of most Hacker News folks, but so be it.

When you sold the site to Condé Nast they became your customer. Looking after your customers is the number one priority of any business. So it is totally correct that you made sure they received a good product and got the best from it.

So you're saying from the Fall of 2007 to Fall of 2009 he was a full time hire as opposed to a contractor like it says in the transcript?

Thanks, sorry I didn't RTFA.

Reply on news.ycombinator.com