The one big difference is that you don't get the advantage of the (relatively speaking) easy to understand and code against Map-Reduce API. Programming in MPI requires a bit more low-level fiddling... something like the difference between, say, programming in C vs programming in Java. Which is, I suppose, quite appropriate since most MPI programming is done in C (or using the C bindings from another language).
There are also some Map-Reduce implementations natively written in C++ floating around out there.
Note that if you do something like a Beowulf cluster you still need a distributed file-system to serve the role HDFS plays in the Hadoop world.
There's also this:
https://gigaom.com/2015/02/18/google-open-sources-a-mapreduc...
but I haven't used it and don't know much about it.
[1]:https://en.wikipedia.org/wiki/Beowulf_cluster
[2]: http://www.rocksclusters.org/wordpress/
[3]: https://en.wikipedia.org/wiki/Message_Passing_Interface