MapReduce vs. SQL: It’s Not One or the Other | Hacker News Reader

MapReduce vs. SQL: It’s Not One or the Other

gigaom.com

9 pointspeter12311 comments

I thought MapReduce and SQL were totally separate tools to solve totally separate problems...

Yes. The actual paper ( http://news.ycombinator.com/item?id=562324 ) compares MapReduce to parallel relational database management systems, not "SQL".

Also, if this gets more points than the earlier link to the paper itself I will be sad.

EDIT: sadness.

Also, if this gets more points than the earlier link to the paper itself I will be sad.

It's all in the headline. If the paper was titled "Researchers confirm it: MapReduce doesn't scale" (and it if wasn't a PDF) it would have a much higher score.

You are absolutely correct. MapReduce is chiefly a framework for performing distributed computation. RDBMSes, colloquially referred to as "SQL databases", are for data storage and (parallel, transactional, ...) access. If you want to compare SQL databases with some Google technology, probably the closest thing would be http://labs.google.com/papers/bigtable.html Bigtable. MapReduce doesn't even come close to being anything comparable.

MapReduce and parallel SQL implementations are perfectly comparable, as the paper itself makes clear: both are for doing large-scale data analysis. You can use RDBMSs for other things, but it is perfectly feasible to compare shared nothing massively parallel SQL implementations with MapReduce, because they are often used for similar purposes.

You can write a paper comparing Perl with MySQL, and if you put enough effort into it you may even conclude that MySQL has better performance, but this doesn't make the comparison any more meaningful.

How is the comparison not meaningful? MapReduce and parallel SQL implementations (e.g. Greenplum, Teradata) are competitors, and are typically used for precisely the same task (large-scale data analysis). They obviously differ in some ways (which is what makes a comparison interesting), but your suggestion that "MapReduce doesn't even come close to being anything comparable" is just uninformed.

I'm surprised by the direction this conversation is taking. I assumed the confusion came from a misunderstanding of what MapReduce was, since it's a much less known technology. My comment was intended to clarify this confusion. I didn't expect a disagreement about what a relational database is.

I've never heard of Greenplum and Teradata. I had however used a number of RDBMSes for five years at three different jobs. I read a book and took a class on relational databases. Persistent storage and concurrent access were central features of any database I've ever used or learned about. The databases the authors of the article use seem to be of the same kind. For example, they first load the data into a database and then use SQL to query it.

There are no equivalent concepts in the MapReduce framework. It doesn't make any sense, for example, to talk about loading data into MapReduce. A map reduction is typically a process that takes input from some place in some form (could even be a relational database), performs some computation on it, and writes the output in another (possibly identical) form. I don't see how you can say the two are competitors, any more than Perl is a competitor to MySQL.

I actually know quite well what MapReduce is typically used for, and I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks.

I've never heard of Greenplum and Teradata.

They are parallel databases, which is what the article is comparing MapReduce against. The basic idea is that you partition the data over a set of machines, and similarly divide and parallelize query processing.

MR is basically a specialized parallel DB that executes a single query of the form:

    SELECT map(...), reduce_agg(...)
    FROM ...
    GROUP BY map(...)

Again, MR's architecture differs in various ways from the design of a typical parallel DB, but they are certainly comparable.

It doesn't make any sense, for example, to talk about loading data into MapReduce

MR input data is typically stored on a distributed filesystem (e.g. HDFS, GFS). Loading the data onto that filesystem is analogous to loading the data into a parallel DB; in both cases, the input data is partitioned over a set of nodes. For good performance, you want a copy of the data to reside close to where the subsequent computation is going to take place. But certainly MR and DBs differ in this respect: a DB typically assumes it "owns" the data, whereas MR is more amenable to processing data stored in an arbitrary external format. This difference is not fundamental, though.

I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks.

You would be mistaken, then. If you want to do analysis on large volumes of data using a cluster of machines, both parallel DBs and MR are viable choices. The whole point of the paper is to take a set of queries, execute them using both Hadoop and parallel DBs, and compare the resulting performance and user experience.

I’ve heard Google itself uses SQL, MapReduce and/or sharding depending on the task.

Wow, that's a lot of cluelessness in a single sentence.

This was my biggest take-away from Web 2.0 Expo. I thought it was a replacement database. But the guys at cloudera did an excellent job of explaining that Hadoop/MapReduce can be used for doing analysis of huge amounts of data, and prepping that data to be exported in a relational database.

Reply on news.ycombinator.com