Getting Creative with MapReduce | Hacker News Reader

Getting Creative with MapReduce

sritchie.github.com

58 pointssritchie7 comments

My approach to this is to take the complicated bits of the mapreduce and put them in a separate class. Then I do a combination of two things as appropriate:

1. Hook it up to a debug server that fetches from the same datastore as the mapreduce, then test it on some keys that I'm interested in.

2. Test it like any other class.

The only awkward part of this is abstracting out the output calls, which I usually do by passing in a "handle some data" callback that outputs in the mapreduce and dumps some pretty html in the debug server.

The great part about this is that if the mapreduce ends up being something important, you already have the tools to introspect its internals on data you are interested in.

And if it ends up being un-important, what then? My contention here is that if you can't pursue small ideas that might end up being unimportant, you're limiting yourself creatively in a big way.

These methods exemplify the sort of complexity I was getting at in the blog post; how many tests do you end up writing if you need a full debug server?

Midje and Cascalog make it possible to do test-driven development at large scale. Is anyone else in MapReduce able to practice TDD? Really curious here.

I find writing a debug server easier than writing tests, but maybe that's just me. :) Especially for speculative things, the notion of whether the code is "correct" could change from moment to moment, so I find it much more useful to look at what it's doing than to test if what it's doing matches a spec that doesn't exist.

Hey, if it works, no problems here.

Personally, I've found it really helpful to be able to think of and test queries as pure functions (without side effects related to data storage). Composable, pure queries keep our code base really small, and allow us to code and test features in hours.

I end up writing and testing a lot of queries that I never use in production, but it's this play that allows the good stuff to bubble up.

I use Pig to write many of my Hadoop Mapreduce jobs and test my Pig script on a small dataset using Pig Grunt. Pig 0.9 has a lot more debug information as well.

The Cascalog abstraction layer fixes this issue by separating logic from data, allowing you to play creatively at massive scale.

I just checked out Casacalog and I like what I see, although I have yet to try it out myself. Does anyone know of something similar that would work with Scala as well?

Nice.

Check out another similar clojure library called "MR-Kluj" that you can use to write Hadoop MapReduce jobs in Clojure: https://github.com/cheddar/mr-kluj

Reply on news.ycombinator.com