On the Death of Map-Reduce at Google | Hacker News Reader

On the Death of Map-Reduce at Google

the-paper-trail.org

47 pointsaburan2811 comments

Previous discussion: https://news.ycombinator.com/item?id=7947782

> _So it’s no surprise to hear that Google have retired Map-Reduce._

As a Googler my reaction to this statement is "LOL". We still do a ton of map reduces.

I'm not sure if you're allowed to talk about it, but...

I remember a talk by Rob Pike where he mentioned that Sawzall was one of the first major MR users within Google. But IIRC, Sawzall has been retired at Google some time ago, and the successor to it was developed in Go. Does the successor follow the same or similar principles in terms of parallelization, or did you follow another approach?

The successor to Sawzall is basically a framework for writing mapreduces in Go.

Do new applications make use of it, though? "Deprecated" might be a better term to describe it.

Yes, new mapreduces are written every day.

MR is actually growing at Google still and heavily used by legacy systems and experienced engineers.

Compared to newer frameworks like those described in the FlumeJava and MillWheel papers, MR's growth is flat.

This is also happening in the Hadoop ecosystem too: if you're writing JavaMR by hand, you're probably spending more time and writing less efficient jobs than what you might get by an optimized pig/hive job with tez under the hood. Or through something in the Cascading or Crunch family, which provides useful abstractions on top of MR or other execution engines.

Then there are also a lot of tools popping up that take some if the use cases that were shoehorned into MR which are more natural outside, like ML/iterative computation through Spark.

MR isn't dying inside or outside Google, it's just being abstracted away.

I love how the old research into this is just ignored. There is a ton of work which has showed for 20 years or more that map-reduce is just a subset of interesting parallel computation patterns. eg: http://pubs.doc.ic.ac.uk/parallel-skeleton/

I'm kind of late to this map-reduce party in terms of large scale data processing. What is the paradigm shifting towards. The point about MR being a trivial subset seems rather obvious to me. Hadoop for instance is still in a phase of getting a lot of adoption yet is being implied as being a dead man walking. Raises a lot of very interesting postivie questions for me

The link makes more sense without the anchor:

http://the-paper-trail.org/blog/the-elephant-was-a-trojan-ho...

Businesses are already mixing map reduce with other approaches. On the Hadoop platform we have Spark, YARN, Storm etc which all work on the existing infrastructure.

Map reduce however is still perfectly suited for many batch processes.

Reply on news.ycombinator.com