It's usually more efficient to use distcp to copy the data to ephemeral storage:

    hadoop distcp s3n://<id>:<secret>@bucket/input_data /input_data
    hadoop jar myjar.jar org.myorganization.module.jobname /input_data /output_data
    ....other steps....
    hadoop distcp /output_data/ s3n://<id>:<secret>@bucket/output_data

Doing it this way, all spills/intermediate steps live on (very fast) ephemeral storage. Additionally, the distcp command is pretty fast since it is a distributed copy - all nodes will saturate their connections on the copy.

Also, the authors advice to "make processing faster, decompress it, split it in to lots of smaller files" is probably not the optimal way to do it. Hadoop isn't great at storing lots of small files.

http://www.cloudera.com/blog/2009/02/the-small-files-problem...

I'm guessing that he gets a speedup because he is using raw s3:// as a filesystem rather than s3n://. If I understand the hadoop s3 connection, this means that hadoop will use a single connection to pull a single file - so a single 10GB file will max out a single node's network connection and the rest will go unused. Splitting the file up into 1 million line blocks lets multiple nodes pull simultaneously. But storing files using the s3n:// allows this anyway, and results in large files being split into 64mb blocks. It also allows you to store files larger than 5GB, or whatever the s3 file size limit is.

Now, a question I'd really like to know the answer to. What advantage is there to using EMR vs Whirr?

[edit: adjusted language - upon rereading it seemed harsh,which was not my intent. ]