Hacker News Reader

Comment by dwhitney | Hacker News Reader

Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more along the lines of, "wow, look at how old skool Facebook is!"

View on HN

You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.

There is this article:

http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin...

Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one.

They are (obviously, this being Yahoo!) not on Amazon though.

Define bigger? More nodes yes, but less cores, RAM and disk space.

From the article:

That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB.

Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count.

[1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...

Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.

Reply on news.ycombinator.com