MapReduce explained | Hacker News Reader

MapReduce explained

joelonsoftware.com

47 pointshuherto10 comments

Java functors are ugly. I've seen an example where every column of a database had a corresponding class. This facillatated the use of functors.

Java has another example to teach. JavaBeans can be called transparently via remote procedure calls. When developers discover this feature, they write everything as JavaBeans on the assumption that it increases scalability. Instead, it leads to obscene implementations with trivial operations requiring hundreds of remote calls within a server cluster. Understandably, this adversely affects performance.

So, don't prematurely optimise. Use map and reduce sparingly. You wouldn't use RPC to make a string uppercase. Likewise, you wouldn't sum an array of 10 integers using reduce.

Of course you wouldn't uppercase a string using RPC that's what you have map reduce for... split that string into chars send each node a single char to uppercase and combine the results you get back.

And they said programmers have a tendency to make things overly complicated...

As one my teachers used to say: "It is better to use a bad language well than a good language badly" The blame is not so much the language but the programmers.

This is a great insight, one that I can unequivocally agree with, and it led me to an insight of my own. Modern-day teachers have the cause-and-effect backwards. Instead of focusing on improving the quality of the programmers, they are focusing on developing a language which is impossible to use incorrectly, no matter how bad that language is. Java can certainly be seen as a step in this direction.

You wouldn't sum an array of 10 integers using reduce... you'd most likely use fold.

Spolsky was talking about the power of abstracting away for loops. Functional programming languages do this all the time, with primitives like map, fold, filter, etc. And this happens without any toll on the programmer's mental burden or the machine's performance.

Nope, I would use sum. Python, Ruby, Lisp, and countless others already have it defined. If I happened to be using a language that didn't define a sum function, I would write one.

A more in-depth look at the same stuff:

http://www.youtube.com/results?search_query=Cluster+Computin...

The most interesting thing in my opinion is that MapReduce implementations has been made with c++ (Google) and Java (Hadoop/Yahoo). And not with a functional programming language.

Theres a rather good explanation of MapReduce in the book "Beautiful Code". (Its just a rather good book, period.)

I think the book is very hit or miss from chapter to chapter.

Reply on news.ycombinator.com