Crunch – Go-based toolkit for ETL and feature extraction on Hadoop

github.com

37 points·jondot··11 comments

That's an unfortunate name for a Hadoop-related project.

https://crunch.apache.org/

Looks cool, how's the performance in Go compared to other tools like Python or Ruby streaming?

I'm getting great performance out of Go. This is the main reason from moving off my previous JRuby toolkit (also open-source)

https://github.com/jondot/hog

Eventhough streaming is 30% slower, Go's speed makes up for that and gives me up to 6x speed than the JRuby based solution.

The Ruby/JRuby solution has its place though, you have tens of thousands of valid ruby gems to use and it makes a very rapid development experience.

What about compared with Java, a language with a proper native compiler?

Unfortunately the standard Java map/reduce interface will win out because the IPC is more efficient. But then doing ETL that way becomes clumsy and painfully slow. The idea is the use rapid tools to do the same job. Streaming is then one step higher across the happiness factor because you just use

Unix pipes to test your solution (and you can keep piping other tools as well as pipe Crunch processors over and over again)

Is this useful for load targets other than Hadoop? RDBMS for example?

Actually yes! Because I'm using Pig to "abstract" I/O you can use Pig to load data from anywhere (any format), it streams it into Crunch, and then again, Pig to save data to anywhere (any format).

Tested it right now, so easy to use and works great! Great stuff

Enjoy! :)

Do you have an example for doing ip2location of some kind? We do it today not optimized and it will be helpful

Sure, will prepare something and push it soon

Reply on news.ycombinator.com