I ran a web app with identical pageview numbers using Ruby on Rails and EC2. My server costs were 10x cheaper than what reddit is paying.
Scaling is caching and architecture, not writing your app in Java.
Scaling is caching and architecture, not writing your app in Java.
I'm also not sure what makes a Facebook game's domain more "widely cachable". 100% of users are logged in. The vast majority of actions taken are changing state. Any app touchpoint is writing to the database. Page caching is nearly impossible.
It's a lot less different than you think.
You obviously know how to scale an app if you were pulling 16 million pageviews a day, and I don't intend to discount that at all. I just mean to point out that while fundamentally the same problem, reddit has to deal with a version of that particular problem that most applications don't begin to approach.
I never understood this. Does reddit really need to spend the capital making sure I see a stranger's upvote the moment it occurs? A 60 second delay to refresh pages in batches seems perfectly reasonable. Perhaps with a client side script to mark my own upvotes so the system doesn't look like is losing my selections.
Mibbit is written in Java (Custom written framework and server), and handles traffic just fine. Java is insanely efficient for network IO.
It's an apples to frogs comparison, but if you measure 'page views' then Mibbit does bajillions, on a handful of servers.
Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter.
I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage.
I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL.
I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%.
Total hosting costs were ~$2,000 a month.
Facebook giveth and Facebook taketh away.
That's cute, but was your web app as hard to cache as Reddit?
Scaling a fully static website is trivial, and the more dynamism you include, the harder it becomes.
There is barely anything static on a logged in Reddit user's page.
USERCLASS = SHA1(SUBREDDIT1 XOR SUBREDDIT2 XOR SUBREDDIT3)
For logged in users, you're still going to need the voting status for the current user on every single submission as well as that submission's hidden status for the current user to decide whether or not a submission should be displayed in the listing.
I believe they almost never hit the DB directly, so these are probably recached immediately (or submitted to both the cache and the DB at the same time), but that still means quite a lot of traffic.