They don't have to check 1.7 million users to see if they have @replies on, if they maintain a list with "@reply subscribers", there's little overhead as far as I can see. If a something rarely changes but is checked a lot, it's a pretty good idea to have it pre-calculated.
Edited: "No overhead", was a bit to absolute thanks jpwagner.
Re-Edited: Oh look there was a commenting feature on the blog, was pretty hard to find.
instead of just one flag link on an HN post, there should be
flag as inappropriate vs.
flag as stupid
*edit: make that "flag as waste of time"
In fact, what you've pointed out is that building an implementation to address the 3% case is straightforward which was the point of my post.
PS: My blog does have a commenting feature. In fact there are 3 comments in response to the post.
Or if you're going to hide them, at least put the view comments link at the bottom of the post where I'd expect the comments to be.
Person Y member of 97%
Person Z member of 3%
Person W member of 3%
________________________
Person_A_@_List: X, Y, Z
Person_B_@_List: X, W
________________________
Person_A writes: @Person_B you be cool!
Received by X and Z...not Y or W
With the new system, say user Foo writes "@Bar lol me too!". Then Twitter can take Foo's follower list, join it with Bar's follower list, and send the message to everyone in the resulting list. Relational databases are very good at joins.
On the other hand, with the old system, they'd have to do a deep inspection of the record for each of Foo's followers to know if they should send the message to that follower. Relational databases are much less good at this.
But, as you and others have pointed out, if the number of users that use the "all @-replies" feature is really so small, it would be fairly inexpensive to cache the list of all of Foo's followers who use that feature, and join them in as well. I don't know why they don't do that -- maybe it adds up (like, if only 3% of users use the feature, but those users follow a lot of other users, they'll each end up in a lot of other users' caches).
(Following up: A trusted source confirms 3%)
Note, btw, that the number they seem to be going for is the above -- e.g., "according to the accounts database, only 3% of users ever changed this setting", _not_ "only 3% of users are claiming to care about this now that it's been removed".
It's so easy to sit here and play armchair engineer and diagnose Twitter's (mostly past, btw) problems with a nauseating mix of hubris and ignorance.
That, and the pointing out of quite natural questions which Twitter has failed to address in its explanation of how it is acting. I mean, once you know what you are doing, being able to satisfy people asking ignorant and hubristic questions is a pretty sweet bonus.
I'm afraid it's a nauseating mix of having operated real pub-sub systems such as TIBCO Rendezvous on major stock exchanges.
What Twitter does is not difficult and they do it badly.
You started out with a condescending post which implied that anyone who'd stayed awake in CS 101 would find Twitter trivially easy to scale.
Then you switched course to claiming that you know how to do this because of experience working a gigantic installation.
Or are you asserting that the average CS 101 project consists of running the production messaging systems for major stock exchanges?
He didn't claim his experience proved HE could do it. He didn't imply that you NEEDED his experience to do it. He was calling you out on your claim by showing that he had done it, and so knew from experience that it wasn't hard. You're the one moving the goal posts.
Note: I actually don't have an opinion on whether that experience is relevant to twitter's situation, but it certainly wasn't a bait and switch.
The fact that he went from "CS 101" to "major stock exchange" as the barometer seemed a little fishy to me, ya know?
And how many stock symbols are there? 5500 on NYSE, 4500 on NASDAQ.
Updates per second? CSCO had a few hundred thousand updates per day.
How is the data persisted? It is typically only logged, and not retrieved in realtime.
Unfortunately, Twitter doesn't match this problem particularly well.
Delicious, which I did write, had many similar structural issues. Tags, /network, etc.
Finally, how much do you know about Twitter's architecture and do you know the definition of hubris?
The real question is how much VC has Twitter wasted reinventing the wheel (and making it square) when they could have just bought an R/V license?
"Not invented here" is the definition of hubris.
Every time someone wrote a reply Twitter... this was the most expensive work the database
Sorry, I am going to laugh at people having scaling issues when they use a database like that on a messaging system. There was an essay by PG where as soon as he heard they where using Oracle (or C?) he instantly knew to ignore them for much the same reason.
Anyone doing this over without a point to prove would start with http://www.rabbitmq.com/
Not a case of magic, just the right tool for the right job. You could code a web app in C, but why would you? Making twitter in Ruby (and to a lesser extent Scala) is the same thing.
I clicked the profile link to your blog, and it looks like you have a lot of Ruby experience. Compare the syntax for message passing between Erlang and Ruby.
id ! {my_fun,Args},
Thats it. Passing messages between pids on nodes across a network isn't much harder.
Compare the difficulty of parallel computation and storing data in parallel across many systems reliably between Erlang and Ruby. Doing that stuff in Erlang is just as easy as doing CRUD in RAILS, because the tools are designed to make it easy.
The downside is that Erlang is pretty bizarre and hard to learn if you've never coded in a functional language - you only get to set values ONCE, etc. Thats a critical feature for debugging large pools of processes working together, but it sure is strange. Which is why you wouldn't use Erlang to do a simple CRUD web app, you'd use RAILS.
Seriously... its an entire language based around passing messages between lightweight pids working in congress, across many systems, with built-in fail-safes for any failures, and a rock-solid distributed/redundant data store baked in.
In this situation its just about as close to a silver bullet as you will ever see. Erlang would make writing twitter's back-end FUN.
A dozen crazy Swedish mad scientists slaved over a language for years to enable creating scalable applications like twitter trivial, only to be totally ignored by most everyone until quite recently. And still twitter picks something else for the rebuild.
'Heavy metals poisoning' would seem to be a more apt metaphor than 'silver bullet.'