What it really means when someone says ‘Hadoop’

linkedin.com

3 pointskathironline3 comments

I still don't understand the value add of MapReduce (in it's various implementations, including Hadoop) versus Clustered SQL. I see articles like this one that seem to imply that there is a niche for MapReduce for tasks like simple string search on super large data sets: http://gigaom.com/2009/04/14/mapreduce-vs-sql-its-not-one-or.... It just seems odd that people are so quick to throw away 40+ years of research on how to structure relational data and how to optimize queries against large data sets.

Hadoop is free for 400 (or n) computers. Let's say you need 100 computers (full time) to process the equivalent clustered query in Oracle, given the time needed to get the formatted data. How much do the licenses for that cost? (And if it's a free system, you're still going to have to do an ETL job, so why not ETL it to a free solution?)

You need to process a couple terabytes of data, but you only need to do it once a month. However, even the highly tuned SQL will bring the system to its knees. So, 90% of the time really expensive servers are idling because of these peak processing needs. Once a month. (And Oracle isn't going to be available on a cloud service.)

So, yeah, if someone had a free/cheap SQL solution for these kinds of ETL datasets that was able to handle these embarrassingly parallel problems, people might be interested. Perhaps you are aware of solutions I am not.

Can a mod change the url to the unframed version?

Reply on news.ycombinator.com