GNU/Parallel changed my life

unethicalblogger.com

76 points·recampbell··8 comments

"Unlike xargs however, Parallel lets me make use of the many cores that I have access to [...]"

Unlike xargs? That's just plain wrong, see xargs' -P flag.

xargs can't distribute tasks to remote CPUs though.

Oh man ! I wish someone had told me this before. Back when i did research for my BSc thesis, I had an 24 core machine and wrote a script to create 24 bash scripts to split up my task which i would than run in parallel.

Sometimes I have the felling that I am wasting time when I check HN too often, but stuff like this makes me think otherwise.

You could have also used "make -j", depending on your specific task.

This is sometimes true.

I think that Parallel is more interesting as an instant server-farm for "taco bell programmers" -- q.v. http://teddziuba.com/2010/10/taco-bell-programming.html and http://news.ycombinator.com/item?id=1818816 for the HN discussion -- as per Mr. Dziuba's definition.

Like, async spidering with parallel is a one-liner, but comparatively, have you seen how much absurdly syntaxed pycurl boilerplate you need? It's crazy. I for one will take the simpler method any day.

Previous discussion on GNU parallel:

http://news.ycombinator.com/item?id=1801186

"Changed my life," a bit dramatic methinks.

That is just going to thrash hell out of your disk unless it's executing on distinct machines with data well spread over a shared filesystem.

Reply on news.ycombinator.com