Maybe I'm missing the part where you can just batch job it and read per line with hadoop / python and the csv lib?

Shoot me an email ( details in profile ) as I'd love to poke at something just for the line on my resume.