GNU Parallel - build and execute command lines from standard input in parallel
savannah.gnu.org
savannah.gnu.org
Examples should be at the top!
(10 years of frustration in that one line message :p)
1. Is single threaded 2. Takes a lot of CPU 3. Is a task that everyone can understand and relate to and which is close to a real world scenario
I have loads of examples meeting requirement 1+2. It is 3 that is the hard part.
Post them to parallel@gnu.org
$ ls *.png |wc -l
3580
$ time ls|sed 's/\(.*\)\..*/\1/'|parallel convert {}.png {}.ppm
ls --color 0.00s user 0.01s system 63% cpu 0.016 total
sed 's/\(.*\)\..*/\1/' 0.01s user 0.00s system 39% cpu 0.025 total
parallel convert {}.png {}.ppm 97.39s user 61.87s system 890% cpu 17.883 total
$ time ls|sed 's/\(.*\)\..*/\1/'|xargs -I {} convert {}.png {}.ppm
ls --color 0.01s user 0.00s system 63% cpu 0.016 total
sed 's/\(.*\)\..*/\1/' 0.01s user 0.00s system 39% cpu 0.025 total
xargs -I {} convert {}.png {}.ppm 93.08s user 47.88s system 38% cpu 6:10.88 totalI wrote a simpler one a couple of years ago (http://code.google.com/p/spawntool/) myself. All it does is read commands from stdin, one per line, and keep a desired number of processes running until all command lines are exhausted. Simple.
I wrote my own because I got tired of all kinds of substitution and quoting issues with xargs. With spawn I only need to generate the shell commands and instead of piping them to bash I pipe them to spawn. Also, this means I can easily review your command line generation with less (so that quotes etc. are good) until I eventually switch to sh or spawn.
echo "file1 file2" | xargs -P 2 gzip echo "file1 file2" | xargs -P2 -n1 gzip
?And the version of xargs that is included with Solaris 10 doesn't have the -P option. In which case, installing gnu parallel is a slightly easier option than installing a different version of xargs.
Convert .mp3 to .ogg running one process per CPU core on local computer and server2:
parallel --trc {.}.ogg -j+0 -S server2,: \ 'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3Standard "./configure && make && make install" outputs errors.
Perhaps the Portfile will have the patches you need to manually compile on OS X:
http://trac.macports.org/browser/trunk/dports/sysutils/paral...
parallel --version
There also seems to be a few more options revolving around job success/failure and how to react -- a) ignore failed jobs and report how many at the end, b) cleanly exit as soon as a job fails and c) stop all jobs as soon as one fails.