Twitter's peak load is only 200 tweets per second | Hacker News Reader

Twitter's peak load is only 200 tweets per second

friendfeed.com

BUT, that's via the firehose, which doesn't include direct messages or twitterers who've locked their feeds. The number also doesn't reflect what has to happen with each of those 200.

Sure it's not impressive if all they have to do is append a 140 char message to a flat file or DB table. It's all the other manipulation that's interesting.

What kind of 'manipulation'? They're basically routing messages, with a protocol change in the middle when the recipients' designated protocol differs (i.e. IM to text message). This can't be all that computationally expensive when the system is engineered right.

Knowing how little activity twitter has really makes them look incompetent in light of the service outages they experienced over the last year. Either Ruby on Rails is really really non-performant or the twitter code-monkies got the system architecture wrong.

The correct metric is probably not "tweets incoming", but "tweets delivered". There's some multiplicative factors that start to bite.

I say "probably" because I'm an outsider guessing, but I'm reasonably confident about that statement. Just not quite 100%.

(In the colloquial sense of 100%, of course; I'm not mathematically 100% sure Twitter exists.)

For starters, there's a few interesting links at http://highscalability.com/scaling-twitter-making-twitter-10...

summary: "Rails and Ruby haven’t been stumbling blocks... The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January"

They're dealing with duplication, consistency, searchability, etc. across distributed storage systems and a variety of service mechanisms (more than just protocols). The 1 000 000 follower user has every message duplicated and cached at multiple layers up to 1000000 times. The message can show up on the web, API, or across any of the protocols, and it has to persist.

They're not incompetent code-monkies, they just guessed way low when they designed the architecture. It looks like the big move was from a CMS model (not at all unreasonable for a "microblogging" service) to a messaging model. In hindsight, targeting messaging to begin with would've saved them some down time, but wouldn't have been practical in the short term.

From http://gojko.net/2009/03/16/qcon-london-2009-upgrading-twitt... it sounds like the insertion number averages around 9600 messages per second. That's avg follower count * avg tweets inbound, and that's only the input.

Edit: real, solid numbers are hard to come by. I suspect it's proprietary.

What you say is echoed by, Every incorrect assumption in this post seems to think that 1 tweet on twitter = 1 database row = "So easy!". You've left out the user fanout! One Obama Tweet = 1M database rows, someplace

Assuming that is true, which I have trouble believing it is, it sounds like they need some help with normalization. I understand the tradeoffs, but it just seems crazy.

Select tweet.* from tweets inner join tweeters on tweeters.id = tweets.tweeterid inner join followers on followers.followerid=loggedinuserid and followers.followee=tweet.tweeterid

Yeah, okay, I know there are performance problems with joins, but there are performance problems with 1,000,000 inserts as well and you could cache the list of followees and do an "in" statement such as: select * from tweets where tweeterid in (cached_comma_separated_list_of_followee_ids)

or whatever to improve select performance.

Thanks for the links. This slideshow was also useful to me as it has a couple conceptual diagrams of what Twitter is doing. http://www.slideshare.net/Blaine/scaling-twitter

I really really doubt that "all they have to do is append" 140 chars" to a table.

I don't see the value in Scoble's argument, if he has one. I don't see the point of comparing the amount of tweets/sec with the rate of Google queries. They are just very different things, in their essence, purpose, way of usage.

It would be more interesting (maybe) comparing the number of Twitter search queries with Google's, at least we would be talking about similar operations.

Yes. I think Google has to do quite a bit more per query than Twitter per message.

and google also has perhaps hundreds of thousands of servers and tens of data centers.

His point is just to provide some measure of scale against a "known beast".

I'm confused... Where is this figure actually coming from? (Other than Scoble saying it.)

According to him, it's the number given by "people who are getting the firehose feed."

http://friendfeed.com/scobleizer/50e673d8/some-stats-from-tw...

That makes the headline seem a bit linkbaity to me. It also makes me question the validity of the data.

Another way to extract this figure is to monitor the ids of tweets like the Twitpocalypse site is doing.

We've been monitoring Twitpocalypse for a while, and the data yields the following chart:

http://img.skitch.com/20090529-xhhnr83xsy67burr7w24kycp4u.pn...

From the slope, we saw about 200 tweets/second, sometimes peaking at 250 as well. The right axis in this chart is the number of tweets per minute. You can see it rising and falling through the day.

Hold on a minute. Are we talking about peak load or average load?

Saying Twitter averages 200 tweets/second is vastly different than peaking at 200 tweets a second.

Am I just being slow?

Edit: Nevermind, Robert is saying that the average is 200 tweets/second during periods of peak load. The headline here makes it seem like the maximum capacity is 200 tweets/second which is misleading.

I think what the data shows is that the average sort of wobbles around a bit. Check out the update to my post for a new image link to see it varying throughout the day.

80KBps, 0.5 billion tweets per month.

(200 hertz) * 140 bytes = 27.34375 kBps

If they ran at peak for 24 hours a day during an entire month. Which isn't what peak means.

MySql can handle that with both hands tied in the back.

No sweat.

Uh, some of those 200 tweets per second are being sent to over a million people. Most are sent to hundreds or thousands.

I don't think that actor guy posts every second, and the average number of followers is way lower than thousands.

There are 13 users with 1000000+ and the top 100 all have over 450000. (http://twitterholic.com/)

But, long tail and all that, I hear average follower count is in the 120 range, you're right there. (http://gojko.net/2009/03/16/qcon-london-2009-upgrading-twitt...)

I've always assumed that most people don't get every tweet SMS'd to them. So most of the time, a tweet isn't sent to anybody, it's just returned when people request the tweets that are relevant to them.

twitter does not

"INSERT INTO tweets VALUES ('me', 'new tweet', DATE())"

what mysql can handle doing a completely different thing is pretty irrelevant

The MySQL load is much higher than 200 Hz. Depending on how they do it, what you want to count as dependent load, and how you want to measure it all, you have a scalar for the average number of followers to update per tweet, work to update indices, etc.

Reply on news.ycombinator.com