Facebook has the world's largest Hadoop cluster | Hacker News Reader

Facebook has the world's largest Hadoop cluster

hadoopblog.blogspot.com

51 pointsDanielRibeiro12 comments

"Sunday, May 9, 2010" just saying... those numbers may be different now.

It was mentioned on Apache Hadoop Goes Realtime at Facebook paper[1], presented on June 12-16 2011.

At least facebook research guys are still advertising it.

[1] ref 16 at http://borthakur.com/ftp/RealtimeHadoopSigmod2011.pdf

Does anyone knows what caused such big spike in size of data around August 09? Did they simply move more/all data into Hadoop or is there another explanation?

Number of machine stayed relatively the same, so it seems really weird that they increased size of data 10000-fold while not even doubling their machine number.

If I was going to guess, I'd guess that they started storing logs of every impression at that point, that'd be a lot of data and Hive's initial use case was parsing weblogs.

Hmm. FarmVille was released shortly before that time, wasn't it? Just a thought.

Except Zynga holds all that information.

Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more along the lines of, "wow, look at how old skool Facebook is!"

You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.

There is this article:

http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin...

Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one.

They are (obviously, this being Yahoo!) not on Amazon though.

From the article:

That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB.

Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count.

[1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...

Define bigger? More nodes yes, but less cores, RAM and disk space.

Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.

Reply on news.ycombinator.com