Hacker News Reader

Comment by phunge | Hacker News Reader

Hadoop is a pile of bad code, a stagnant codebase, crusty APIs and a thick surrounding layer of hype which obscures what it's really like to use. Spark might be better or faster, but mostly what you need to beat Hadoop is to make something practical, which lets developers be expressive rather than wrestle with overdesigned nonsense.

I know this because it's my full-time job to actually get stuff done inside Hadoop.

Spark may be a great system, but this article doesn't do much to settle the issue. When you read fluff like "sweeping software platform", "famously founded the Hadoop project", "great open source success stories" and machine learning described as "crunching and re-crunching the same data -- in what's called a logistic regression", it's time to move on.

View on HN

>>Hadoop is a pile of bad code, a stagnant codebase, crusty APIs and a thick surrounding layer of hype which obscures what it's really like to use.

This is where marketing and branding becomes the primary factor influencing adoption, and not technical merit. Hadoop gathered so much momentum and hype as part of the Big Data buzz in the past few years that it's only now beginning to percolate through to telecommunication carriers and other larger/slower-moving enterprises.*

* I work primarily with wireless carriers; can't say much about broadband, although I'd hazard a guess and say that the majority are only now allocating experimental budgets to see how Hadoop can help them manage their Big Data.

Yeah, I lol'd at that too. Especially given that logistic regression is a tool for statistics, not machine learning (unless you subscribe to the theory that those two fields are one and the same).

> mostly what you need to beat Hadoop is to make something practical, which lets developers be expressive rather than wrestle with overdesigned nonsense.

It would be difficult for an article to convey this. Having used both Hadoop and Spark, practicality and expressiveness are precisely what made me fall in love with Spark. You can do so much with so little code. Don't take anyones word for it, see for yourself. Download the code and run the interactive shell, takes 2 minutes. It was totally mind-blowing for me.

>Spark may be a great system, but this article doesn't do much to settle the issue.

Wired is a mainstream technology magazine.

Reply on news.ycombinator.com