Hacker News Reader

If you're just looking to parallelise an operation over some file then GNU Parallel is an fantastic tool as well. On several occasions recently I've combined the split command and Parallel to break a large CSV file up into smaller chunks, and then run a Ruby process on each of those chunks.

Parallel is apparently also able to distribute a command over several hosts using SSH, although I've not tried that one.

View on HN

Parallel is awesome. The only problem I have with it is that it doesn't support data localization. If you have 500 GB of data you don't want to copy it to a machine to run a command on it. Better to have the data stored on your cluster directly, which is what FileMap does.

Distributed over SSH is awesome. Parallel can push the input files, run the command and pull the output (and then cleanup) all by itself.

Reply on news.ycombinator.com