Hacker News Reader

Comment by typon | Hacker News Reader

Jeff Dean did not invent map/reduce

Jeff Dean and Sanjay Ghemawat developed the system called MapReduce at Google and published a paper about it. They didn't invent the map or reduce functions, nor is the MapReduce system they published really about map or reduce functions.

The correct name is really MapShuffleReduce and the most important step is the shuffle, because it's a distributed sort. Of course, they didn't invent distributed sort, but combining these three concepts together in a distributed fashion and running it as a production system is really what was important.

He did: http://cacs.usc.edu/education/cs653/Dean-MapReduce-CACM08.pd...

The key ideas were part of the MPI interface for a decade before Jeff came along and applied the new branding

https://en.wikipedia.org/wiki/Message_Passing_Interface

MapReduce and MPI are very different. MapReduce doesn't use message passing- the map phase reads inputs from sharded files in a separate disk system, applies map to the inputs, and writes out the mapped outputs to temp sort files on a seperate disk system. Then the shuffler sorts those and writes the outputs to the appropriate destination output shards in a seperate disk system, at which point the reducer reads them, applies reduce, and writes the final outputs, sharded by key to a seperate disk system.

The mappers, shufflers, and reducers are all independent of each other, reading and writing from the filesystem, and managed by a coordinator. There's nothing like MPI, other than the use of the Stubby RPC system, which sort of resembles MPI but has completely different distributed communication semantics.

The distributed database systems of the mid 80s such as the Teradata DBC 1012[1] are better prior art for Map/Reduce.

1- https://en.wikipedia.org/wiki/DBC_1012

OK then tensor flow

Reply on news.ycombinator.com