Scaling Twitter: Making Twitter 100x Faster

This particular design seems to be the official one, but twitter and its lackluster architecture has spawned an industry of twitter optimizers. Everyone and their brother is trying to do something about the fail-whale: I mean, "how hard could it be"? the test implementations are usually benchmarked off of 127.0.0.1 and the "lab" results are, theoretically, impressive.

A similar thing happened when reddit switched to python from common lisp; a flurry of reddit clones in Lisp, each smaller/faster/"eleganter" (pick two) than the other.

You'd be surprised how much one can run off one server should the application not need a lot of reads.

Every single user page should be able to be cached until the user posts again. Likewise, the page of any post should be able to be cached infinitely. They don't run their messaging queue off their database I'd assume.

Really, Twitter doesn't do relational data which makes scaling a good bit easier. The small amount of relational data that they do could be easily denormalized and pushed into memcached. The problem that a lot of web apps face is that it's hard to denormalize because so much of the data is relational. The only relations that I can think of is Person follows Person and Post belongs to Person and Post replies to Person.

So, it's not hard to denormalize that in a way that you can push into memcached rather than having to hit the database much.

I'm not saying that's how I'd do it if I were building Twitter, but with minimal relations and a small amount of data it's easy enough to keep the MySQL store in memory for log(n) lookups from memory and use memcached for O(1) lookups for the majority of stuff.

I think this is outdated. It says 250k users. If aplusk has over a million followers this number can not be correct.

Well, the article explains that their mistake was that they tried to use external software for monitoring and scaling instead of their own, and they switched numerous times because they almost worked, but not quite, until they built their own.