Why Twitter's Engineers Hate the @replies feature
25hoursaday.com
25hoursaday.com
The way it is now (from my understanding) is that for every reply they have to check if somebody is following the person that is being replied to. How can this be more efficient than simply sending the reply-tweets to all followers (which they have to do anyway with all the other tweets that are not replies)?
Considering they've just recently started turning #hashtags into links, I would not at all be surprised if @replies were birthed in a similar manner.
They don't have to check 1.7 million users to see if they have @replies on, if they maintain a list with "@reply subscribers", there's little overhead as far as I can see. If a something rarely changes but is checked a lot, it's a pretty good idea to have it pre-calculated.
Edited: "No overhead", was a bit to absolute thanks jpwagner.
Re-Edited: Oh look there was a commenting feature on the blog, was pretty hard to find.
instead of just one flag link on an HN post, there should be
flag as inappropriate vs.
flag as stupid
*edit: make that "flag as waste of time"
In fact, what you've pointed out is that building an implementation to address the 3% case is straightforward which was the point of my post.
PS: My blog does have a commenting feature. In fact there are 3 comments in response to the post.
Or if you're going to hide them, at least put the view comments link at the bottom of the post where I'd expect the comments to be.
Person Y member of 97%
Person Z member of 3%
Person W member of 3%
________________________
Person_A_@_List: X, Y, Z
Person_B_@_List: X, W
________________________
Person_A writes: @Person_B you be cool!
Received by X and Z...not Y or W
With the new system, say user Foo writes "@Bar lol me too!". Then Twitter can take Foo's follower list, join it with Bar's follower list, and send the message to everyone in the resulting list. Relational databases are very good at joins.
On the other hand, with the old system, they'd have to do a deep inspection of the record for each of Foo's followers to know if they should send the message to that follower. Relational databases are much less good at this.
But, as you and others have pointed out, if the number of users that use the "all @-replies" feature is really so small, it would be fairly inexpensive to cache the list of all of Foo's followers who use that feature, and join them in as well. I don't know why they don't do that -- maybe it adds up (like, if only 3% of users use the feature, but those users follow a lot of other users, they'll each end up in a lot of other users' caches).
(Following up: A trusted source confirms 3%)
Note, btw, that the number they seem to be going for is the above -- e.g., "according to the accounts database, only 3% of users ever changed this setting", _not_ "only 3% of users are claiming to care about this now that it's been removed".
It's so easy to sit here and play armchair engineer and diagnose Twitter's (mostly past, btw) problems with a nauseating mix of hubris and ignorance.
That, and the pointing out of quite natural questions which Twitter has failed to address in its explanation of how it is acting. I mean, once you know what you are doing, being able to satisfy people asking ignorant and hubristic questions is a pretty sweet bonus.
I'm afraid it's a nauseating mix of having operated real pub-sub systems such as TIBCO Rendezvous on major stock exchanges.
What Twitter does is not difficult and they do it badly.
You started out with a condescending post which implied that anyone who'd stayed awake in CS 101 would find Twitter trivially easy to scale.
Then you switched course to claiming that you know how to do this because of experience working a gigantic installation.
Or are you asserting that the average CS 101 project consists of running the production messaging systems for major stock exchanges?
He didn't claim his experience proved HE could do it. He didn't imply that you NEEDED his experience to do it. He was calling you out on your claim by showing that he had done it, and so knew from experience that it wasn't hard. You're the one moving the goal posts.
Note: I actually don't have an opinion on whether that experience is relevant to twitter's situation, but it certainly wasn't a bait and switch.
The fact that he went from "CS 101" to "major stock exchange" as the barometer seemed a little fishy to me, ya know?
And how many stock symbols are there? 5500 on NYSE, 4500 on NASDAQ.
Updates per second? CSCO had a few hundred thousand updates per day.
How is the data persisted? It is typically only logged, and not retrieved in realtime.
Unfortunately, Twitter doesn't match this problem particularly well.
Delicious, which I did write, had many similar structural issues. Tags, /network, etc.
Finally, how much do you know about Twitter's architecture and do you know the definition of hubris?
The real question is how much VC has Twitter wasted reinventing the wheel (and making it square) when they could have just bought an R/V license?
"Not invented here" is the definition of hubris.
Every time someone wrote a reply Twitter... this was the most expensive work the database
Sorry, I am going to laugh at people having scaling issues when they use a database like that on a messaging system. There was an essay by PG where as soon as he heard they where using Oracle (or C?) he instantly knew to ignore them for much the same reason.
Anyone doing this over without a point to prove would start with http://www.rabbitmq.com/
Not a case of magic, just the right tool for the right job. You could code a web app in C, but why would you? Making twitter in Ruby (and to a lesser extent Scala) is the same thing.
I clicked the profile link to your blog, and it looks like you have a lot of Ruby experience. Compare the syntax for message passing between Erlang and Ruby.
id ! {my_fun,Args},
Thats it. Passing messages between pids on nodes across a network isn't much harder.
Compare the difficulty of parallel computation and storing data in parallel across many systems reliably between Erlang and Ruby. Doing that stuff in Erlang is just as easy as doing CRUD in RAILS, because the tools are designed to make it easy.
The downside is that Erlang is pretty bizarre and hard to learn if you've never coded in a functional language - you only get to set values ONCE, etc. Thats a critical feature for debugging large pools of processes working together, but it sure is strange. Which is why you wouldn't use Erlang to do a simple CRUD web app, you'd use RAILS.
Seriously... its an entire language based around passing messages between lightweight pids working in congress, across many systems, with built-in fail-safes for any failures, and a rock-solid distributed/redundant data store baked in.
In this situation its just about as close to a silver bullet as you will ever see. Erlang would make writing twitter's back-end FUN.
A dozen crazy Swedish mad scientists slaved over a language for years to enable creating scalable applications like twitter trivial, only to be totally ignored by most everyone until quite recently. And still twitter picks something else for the rebuild.
'Heavy metals poisoning' would seem to be a more apt metaphor than 'silver bullet.'
Ultimately, though, I'm sure they'll figure out a way for this to work -- especially as it only requires "3%" of users to get @unacquiantanted-replies the same way they get broadcast messages.
One approach: split accounts into two inner accounts, for example, aplusk and aplusk(all). Only the original name is ever publicly shown -- though the account's public page shows the all tweets, as now. However, which one people actually 'follow' depends on their 'show all @unacquianted replies' setting. Thus every aplusk(all) tweet is handled as a broadcast, because all those followers -- only 3% of the total -- prefer it that way.
That would be easier for the twitter engineers, but thats not what 97% of twitter users want. The author doesn't acknowledge this.
That 97% only indicates that most people don't change the defaults; it says nothing about what Twitter users want.
In this case it's particularly misleading, since we're talking about an option to turn off silent data loss. What would make someone track down a setting related to a problem they don't know they have? A more honest representation would be: "A small number of our users were aware that we filter @replies by default, some were happy about it and a few (3% of total users) changed their settings to remove the filtering."