nchammas

Nathan Marz (of Apache Storm fame) also wrote about this topic recently:

"Now, I don't want to comment on the effectiveness of open floor plans for fields other than my own. But for software development, this is the single best way to sabotage the productivity of your entire engineering team."

http://nathanmarz.com/blog/the-inexplicable-rise-of-open-flo...

That may well have been true a year ago, but it is not true as of a few weeks ago [1], and hasn't been true since around Spring of this year.

[1] http://youtu.be/zW0Pqfb8ij0?t=3m50s

> I would love to see MR and Spark compete on the exact same hardware configuration.

You may find this benchmark [1] interesting to read.

It needs some updating (a lot has changed since February 2014), but it compares Shark (which uses Spark as its execution engine) to Hive (using Hadoop 1 MapReduce as its execution engine) and a number of other systems.

The benchmark is run on EC2 and is detailed in such a way that it should be independently verifiable. Hive and Shark are run on identically sized clusters, though I don't know if the other details of the configuration were identical.

[1] https://amplab.cs.berkeley.edu/benchmark/

> Spark _requires_ Hadoop to run

This is not correct. Spark uses the Hadoop Input/Output API, but you don't need any Hadoop component installed to run Spark, not even HDFS.

You can -- and many companies do -- run Spark on Mesos or on Spark's standalone cluster manager, and use S3 as their storage layer.

> this whole Spark vs Hadoop debate makes no sense whatsoever

If we talk about Hadoop as an ecosystem of tools, then yes, it doesn't make sense to frame Spark as a competitor. Spark is part of that ecosystem.

But if we talk about Hadoop as Hadoop 1 MapReduce or as Hadoop 2 Tez, both of which are execution engines, then it very much makes sense to pit Spark against them as an alternative execution engine.

Granted, Hadoop 1 MapReduce is pretty old compared to Spark, and Tez is still under heavy development, but these are alternatives and not complements to Spark.

(Note: In Hadoop 2, MapReduce is just a framework that uses Tez as its underlying execution engine.)

> I just don't understand this constant boasting about Spark, it seems very suspicious to me.

Suspicious how?

I think Spark's elegant API, unified data processing model, and performance -- all of which are documented very well in demos and benchmarks online -- merit the excitement that you see in the "Big Data" community.

Published benchmarks comparing commercial databases are difficult to come by due to the DeWitt Clause [1] in many commercial database licenses.

[1] http://en.wikipedia.org/wiki/David_DeWitt