Hacker News Reader

Comment by gruseom | Hacker News Reader

So the map worker saves map results to its local disk, and eventually a reduce worker does an RPC call to copy that data over the network so it can perform the reduce. What is the advantage of doing this over having the map worker do the reduce part itself?

View on HN

The mapper only has a small fraction of the data for each reduce key.

Couldn't it at least perform the reduce on the portion it does have? i.e. transfer partial reductions from M to R, instead of the raw map results? Presumably there are many problems where this would be a win.

Yes that's sometimes an option, but very often the reduce isn't associative.

Ah, so that's the reason. Do you ever have the problem that the reduce computation, for a given key, is too big for one reduce worker and itself needs to be parallelized? Or does this boil down to the same thing as my previous question - i.e. if the reduce computation isn't associative, you couldn't parallelize it anyway, and if it is then you can just do partial reductions on M.

Yes that is often a problem and is probably my biggest criticism of mapreduce as a framework. Extreme reduce keys often have to be handled differently and in a way dependent on the nature of the problem. Mapreduce doesn't provide great tools for handling it, though I'm not even sure what abstraction you could build that would be comprehensive enough.

The advantage is that you don't need 1:1 ratio of tasks. You could have many mappers and just a few reducers, or the other way around.. depending on how the problem is broken up into tasks.

Reply on news.ycombinator.com