Sorting Petabytes with MapReduce | Hacker News Reader

Sorting Petabytes with MapReduce

googleresearch.blogspot.com

33 pointsabraham16 comments

Google keeps bragging about all their internal capabilities because they want to hire people. But so what? If I had a real reason to, I could fire up a couple thousand machines on amazon and analyze data akimbo.

In a sense, google is worse than microsoft - they really don't share any of their hardcore cs innovations. At least MS is in the business of selling technology. Google is in the business of hoarding it in order to derive competitive advantage in advertising.

I just wrote up an entry about this at http://news.ycombinator.com/item?id=2972368

I disagree with parts of this - Google does release some rather high-quality source code (LevelDB, Protocol Buffers, Chrome/V8), and also provides some well-done research papers.

The entire concept of MapReduce wouldn't exist without Google. And that's inspired a whole host of products. In the same vein, I don't find it unusual that they're NOT sharing the code for MapReduce et al., in that it's proprietary technology. I don't expect Amazon to share the source code behind AWS, and I don't expect Google to share the source code for MapReduce.

Your points are totally valid.

But my point of view is, which company is more relevant to me as a developer and as someone trying to build a startup? And it seems to me that I get a lot more love from Amazon, whether its some nifty new DNS feature or dropping incoming bandwidth charges to 0.

That doesn't change the fact that unlike Amazon, Google is not in the business of selling tools to developers; with that in mind it is impressive what they do for developers despite that.

I don't see that as being a very good comparison. Amazon provides a great service but you have to pay Amazon to get access. Google provides a lot of open source code that you can take, modify, and run on any hardware.

One of my favorite parts of my job is the ability to grab thousands of machines at low priority for the hell of it.

...and without asking permission. "Gee, I wonder how much faster this would be on 5000 machines instead of 2000?" clickicky-click ... "yep, quite a bit faster."

Not permission, but occasionally forgiveness. Task isolation isn't perfect, and backends don't scale infinitely.

As someone at a large corp who just had a 3 day battle to get a new VM with 2 gig of ram and a shared CPU I suddenly feel like crying.

I guess I have no real use for something of this magnitude, but I would still like to get my hands on a seriously large cluster just for the sake of expanding my experience.

I don't suppose anyone knows of a cost effective way of doing so? I have spun up my own Hadoop clusters before but with only 4 machines its not exactly exciting.

Amazon -- you can use Elastic Map Reduce [1] (the easiest way to run a transient Hadoop cluster) or spin up a bunch of EC2 nodes and deploy your own stack. Both methods can provide you access to very powerful hardware [2].

(Disclaimer: I work for Amazon, but not in AWS, and these are my own personal opinions.)

[1] http://aws.amazon.com/elasticmapreduce/

[2] http://aws.amazon.com/ec2/instance-types/

I don't work for Amazon, have hundreds of non-AWS CPUs at my disposal, and still prefer AWS, as long as you don't have to do much I/O.

Once there's a few terabytes of data or more, it's best to be off of AWS, because you'll never be able to saturate your CPUs. Even for smaller datasets of 10s of gigabytes, it's kind of iffy.

Their benchmarks seem a little odd. They claim an 11x speed improvement, compared to using half as many computers three years ago. But you'd expect roughly an order of magnitude improvement anyway, just due to Moore's law.

I'd be really curious to see what part of the speedup is due to software optimizations, i.e., compare the 2011 software with the 2008 software on identical hardware.

From the article: "a large part of the credit goes to numerous advances in Google's hardware, cluster management system, and storage stack".

Moore's law does not suggest order of magnitude performance increase in 3 years.

Moore's law suggests 4x performance increase over 3 years, and they doubled the number of machines, so you end up with roughly an 8x performance improvement from "just" hardware.

No, Moore's law suggests 4x increase in transistor count over 3 years.

Reply on news.ycombinator.com