Hacker News Reader

You should check out mrjob[1]. It wraps the Hadoop streaming API and makes it super easy to write MapReduce jobs in Python. I find it much easier to understand than this implementation.

[1] https://pythonhosted.org/mrjob/

View on HN

mrjob looks nice. If you have a Hadoop cluster. But for "medium sized big data problems" FileMap is a very viable alternative if you have people who know their way around a terminal. Especially if you'd like to have something set up pretty fast. For 500 GB of data setting up Hadoop (steep learning curve; name nodes, jobtrackers, thrift API:s, datanodes, zookeeper and whatnots) is a lot of heavy lifting. Not to mention administering it.

Sure you have Cloudera et al., but I'm still trying to figure out if they really make things easier or worse when it two weeks later comes to figuring out why something is broken, or how to start/install additional Hadoop components.

mrjob has really good integration to Amazon's Elastic Map Reduce, makes it totally painless. I had to analyze 1TB of logs for my thesis and in less than 8 hours I discovered mrjob, wrote my job, and successfully ran it on EMR. Granted I have prior experience with MapReduce, but even to a newcomer, I can't imagine that would add too much time.

If your security policy allow putting stuff into Amazon... ;)

Reply on news.ycombinator.com