Ask HN: Why did Google release their mapReduce algorithm/cluster setup? | Hacker News Reader

Ask HN: Why did Google release their mapReduce algorithm/cluster setup?

4 pointsmonological7 comments

I was just wondering why Google, back in 04, released their map-reduce white paper. They didn't have to. Was it released simply out of good will? Was it done to fend off antitrust litigation? Were other companies already doing the same thing, so it wouldn't matter if it was released? I think they believed that the amount of information that was revealed in the paper wasn't low level or in-depth enough to cause any real damage to Google, even if it were implemented(Yahoo's hadoop). Or am I missing something completely?

View on HN

To engender the feeling that Google is a factory for big new ideas, making it the place every top-flight developer wants to work, and the tech company that everyone talks about when innovation is the subject. I'd say it's paid for itself, and I'd also suggest that it hasn't hurt them. Google has as much of the search market as ever, as far as I know, and the only dents have been due to non-technical factors (Microsoft, and others, modifying the search bar in browsers, ISPs making deals to push other engine, etc.).

I would also add good karma. As in, it doesn't hurt them. I say it was also to showcase their tech and provide more confidence in their later services.

In this particular case the ideas weren't particularly new - when I first heard about Map-Reduce I thought people were talking about MPI's AllReduce function http://www.mpi-forum.org/docs/mpi-11-html/node82.html from the 1995 MPI spec and even then it was nothing new. Teradata and others shipped systems operating on the same principles a decade earlier.

But to answer your question: all large companies need to publish the central technical ideas in one way or another to ensure their own current and future technical staff understand how their systems work (e.g. IBM Systems Journal).

Distributed systems are nothing new (AST's Amoeba goes through some of the mapreduce contortions), but there are very few programmers comfortable thinking that way- you may recall, even SONY had an awful lot of trouble recruiting talent.

It's in Google's best interest to increase the market size of "parallel programmers"- to actually be able to pick from more programmers who can "think" in mapreduce, improves their hiring options.

I doubt this fact escaped Google: mapreduce stands on shoulders of its own giants who faced this problem as well.

Or maybe, like many scientists, they were truly interested in advancing the sum of human knowledge and the state of the art.

I'm just afraid they're being driven by their desire to increase revenue rather than benefiting humanity.

Probably done to get feedback in the area. If you publish a paper on a better map-reduce strategy, Google benefits. The important thing is to seed the research community with the right idea.

Of course, when some researches comes up with a better idea, you headhunt him.

Reply on news.ycombinator.com