Hacker News Reader

The key ideas were part of the MPI interface for a decade before Jeff came along and applied the new branding

https://en.wikipedia.org/wiki/Message_Passing_Interface

MapReduce and MPI are very different. MapReduce doesn't use message passing- the map phase reads inputs from sharded files in a separate disk system, applies map to the inputs, and writes out the mapped outputs to temp sort files on a seperate disk system. Then the shuffler sorts those and writes the outputs to the appropriate destination output shards in a seperate disk system, at which point the reducer reads them, applies reduce, and writes the final outputs, sharded by key to a seperate disk system.

The mappers, shufflers, and reducers are all independent of each other, reading and writing from the filesystem, and managed by a coordinator. There's nothing like MPI, other than the use of the Stubby RPC system, which sort of resembles MPI but has completely different distributed communication semantics.

The distributed database systems of the mid 80s such as the Teradata DBC 1012[1] are better prior art for Map/Reduce.

1- https://en.wikipedia.org/wiki/DBC_1012

Reply on news.ycombinator.com