Google's Secret Weapon: MapReduce

"We're shifting over to a MapReduce-based architecture, just like Google! Google's website is only an interface to a larger architecture. It all ties in to cloud computing and SOA and..."

Naturally we are taking a distributed approach to the problem since hardware is cheap and relatively easy to coordinate via software.

We decided to go our own way using a mixture of some databases and raw file system access with our own application coordinating both the cluster of machines and performing the queries themselves after performing extensive benchmarking of everything.

We looked closely at Hadoop and came to the conclusion that MapReduce was just the wrong architecture for our problem. Its performance was horrible.

The moral of this story is that MapReduce is interesting, but that doesn't prevent you from having to actually test algorithms and make an informed choice. Just because Google's using it doesn't mean it's a panacea.

i got really excited when i was about to find out abt googles secret sauce.. sigh

http://hadoop.apache.org/core/docs/current/mapred_tutorial.h...

I wonder how long that head start will last with a collaborative Open Source project backed by some other big players (that would be Hadoop).

I wrote a comment, part of which you read see here: http://discussionleader.hbsp.com/stibel/2008/12/mapreduce-go...

Then a few hours later my comment has been edited -- they removed a few sentences. (I don't remember the text I wrote exactly, but I said that: 1) the author of this article should have showed his text to at least some CS student, 2) I didn't expect that Harvard Business Publishing could publish articles by such incompetent authors.)

Here we go. They could have deleted my comment, but instead they decided to edit my speech. Why just they don't rewrite everything I said?