Parallel processing with Unix tools

Thanks, I did not know that.

This is useful if you are solving embarrassingly parallel problems that generate lots of log data (e.g. timings for benchmark) and write them to stdout/stderr.

AFAIK there is some guaranteed atomicity when writing PIPE_BUF<512 to a pipe, however then your sub programs shouldn't write more than 512 bytes of log-data at any point, I would guess.

Strongly agree with this!

For 99% of the use cases moreutils' parallel (AKA Tollef parallel) is an equal, or even better, alternative.

"Interesting" thread from 2012 about replacing moreutils' parallel with GNU parallel in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050 (TLDR: skip to message #75).

If the author wants to extract ten seconds of your time once a year, that's just the price of admission. They're not uptight; they're trying to advance their career.

(Info: https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...)

This is a very important problem. We devs love to make tools, and we love to do it for free. But you can't eat idealism. Parallel is a nice tool, but it's probably not as impressive as it seems when it comes to landing a job.

You're free to decry the author and to point out alternatives. It's practically tradition. But I identify with where the author is coming from. How would you feel if you spent a bunch of time on a tool and the top comment blasts it for reasons unrelated to its merits?

I use Linux but I've never used parallels. At first, I thought you were making that up.

But, I looked at your username and realized I have seen you post before - and I couldn't imagine you being dishonest about this. You just don't seem like the type to make this sort of stuff up.

So, curious, I headed to Google.

Sure enough, and for those who don't know, the author wants you to cite parallels when you use it in conjunction with something that is going to be published. Now, to be fair, I've always included citation of the tools involved when I published. However, nobody has ever asked me to directly and nobody has nagged me to do so.

If anyone is unfamiliar:

https://www.gnu.org/software/parallel/

This appears to be the original thread about it:

https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...

It's a bit taboo, I guess. You can pay money and not have to cite it. You can also feed it the --will-cite argument and it won't bug you. Still, it is in probably in poor taste. If nothing else, it is unconventional.

I didn't find a whole lot of complaints about it. The only hyperbolic outrage was, interestingly enough, someone on HN who was quite uptight about it.

I guess Ole is within his rights to do that, but it is certainly unconventional. I wonder if anyone takes his inflated number of citations as a negative against him?

From what I've now read, there don't seem to be a whole lot of complaints. Even the mailing list thread was short and didn't contain any drama. The only drama I found was the post on HN.

Just put this line into ~/.xinitrc or ~/.zshrc or other file that is executed BEFORE you start CLI:

echo 'will cite' | parallel --citation 1> /dev/null 2> /dev/null &

It's GPL, right? Why hasn't anyone forked it with the citation requirement removed?

OK, that has killed all my desire to ever use this software.

Most users won't ever publish anything. Why alienate them?

Author explains background:

https://www.gnu.org/software/parallel/parallel_design.html#C...

I do this pattern a lot

    for x in $(seq 1 10); do long_process file${x} & done; wait

It doesn't take care of not saturating my cpu or anything; when I need to care about that then I try to remember how to use parallel.

The POSIX definition of a "line" is "A sequence of zero or more non- <newline> characters plus a terminating <newline> character"[0]. I don't think it's counterintuitive at all for POSIX utilities to respect that definition.

0. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

The only difference is you have to check the last character in the file to determine if it is a newline. Maybe the last 4 characters if it is Unicode.

I have known situations where the compressed output of pbzip2 was not readable by some .NET library a customer was using (I'm afraid I can't remember which one). Fortunately an alternative multithreaded bzip2 implementation, http://lbzip2.org/, did not suffer from this problem so just in case...

On multiple occasions I found lbzip2[0] to be faster than pbzip2

[0] http://lbzip2.org

Edit: here's a quick sample (file is read from tmpfs on a 4-core i7-5500U):

  $ time lbzip2 < linux-4.13.3.tar > /dev/null
  real    0m17.410s
  user    1m7.799s
  sys     0m0.283s
  $ time pbzip2 < linux-4.13.3.tar > /dev/null
  real    0m30.556s
  user    1m57.557s
  sys     0m2.169s

Also, with modern enough xz-utils you can just use `xz -T` for parallel compression.

If you're zipping multiple files, is it better to pbzip2 each file, or to parallel bzip2 them? You wouldn't want to parallel pbzip2, would you?

Performance of the actual program might be less relevant but it's nice to see it using 1% for itself compared to GNU parallel.

Or any job in general. Also, make 4.x comes with the handy -O flag which makes the output seem "serial" instead of intermixed between processes.

And make -j will execute the rules & build targets in parallel. Once I built a general batch parallel system using make that could pause & resume parallel jobs by using each job's log file as the make target. Then I discovered gnu parallel and scrapped my project.

Very good point. I added a whole section to the article, implementing the counting lines example with `make -j`, which performs just as well as `xargs -P`

You and me, both of us. I once combined rpm -e with xargs rpm -ql to the tune of getting all packages uninstalled from the system in one swift command.

It was a test system so all good. But I'm still not sure what I was thinking.

The simplest use case for xargs is that you have a list of filenames and you want to feed that list to a command that only accepts filenames as CLI parameters:

I.e., 'find' generates a list of filenames.

But 'stat' (for example) only accepts file names as parameters on the command line.

You combine these two using 'xargs':

find -type f -print0 | xargs -0 stat --format="%y\t%n"

And you get an output of human readable modification times, a tab, the filename and a newline.

Note, the "-print0" to find instructs find to output null terminated filenames, and the -0 to xargs instructs xargs to expect to receive null terminated filenames. These switches (they may be specific to GNU coreutils) allow for handling filenames with any possible character in them correctly (i.e., spaces cause no troubles).

For this use case, remembering how/when to use xargs is easy. The more esoteric usages are the ones where "man xargs" is often handy.

It's pretty much 'apply' for shell arguments

https://en.wikipedia.org/wiki/Apply

use it many places instead of a for loop:

for x in ; do <cmd> $x done

is:

echo |xargs cmd

Assuming 'cmd' can take multiple arguments.

If not, use 'xargs -L 1 cmd' to run cmd 1x per arg.

GNU xargs has -1, but -L 1 is portable to xargs in the BSDs and others.. plus it's good to know -L in case you ever need to -L 2, etc.

using echo as a driver is not the best example, but anyhow.

Consumer task will wait on pipe read and the producer will wait on pipe open, so it is synchronized properly.

Need to be aware of the buffer size on the pipe. It can be one of those issues that never comes up until you just cross the threshold, then everything fails ungracefully.

I think it does, try it as:

sleep 5 && (echo hello > test.txt)

and you'll see it creates the file only after sleep ends.

I don't understand what you're trying to say. I would expect these two commands to behave differently:

    (sleep 5 && echo hello) > test.txt
    sleep 5 && echo hello > test.txt

"our intuitive notion of order of operations on its piping" -- I don't know what your intuitive notion is, and there is no piping in your example. Piping would be:

    sleep 5 | echo hello  (this is an odd example, but it works)

I also don't see any race condition in your example. A race would be two processes accessing the same data.

They are redirects, and the way redirects like > and < work is to:

1. open a file

2. change the global process state

3. run arbitary commands, either builtin (echo) or external (ls). External commands require a new process and inherit their process state (descriptor table, where stdout and stdin are conected to) from the parent shell.

4. restore the global process state

5. close the file

If you understand that, it might be clear why the file test.txt is opened before sleep. There's really no other way it can work, given the fact that you can execute arbitrary commands inside the redirect.

I address some related things about redirects here: "Avoid Directly Manipulating File Descriptors in Shell"

http://www.oilshell.org/blog/2017/08/12.html

-----

The more unintuitive thing about shell is that what you wrote is technically better as:

    { sleep 5 && echo hello; } > test.txt

() doesn't mean "grouping" like in other languages, it means subshell. {} means grouping, and it has an odd way of parsing, where ; is required before }, but it isn't required before ). This is because () are operators and {} aren't.

I wrote about this here, "the subshell construct () is confused with the grouping construct { }":

http://www.oilshell.org/blog/2017/02/06.html

-----

EDIT: I will say that I was very confused by the syntax of > and < when I started learning shell. It does seem like ">" means "this goes into that". In fact I think I internalized that from MS-DOS.

But that doesn't explain what these mean, and why they are equivalent:

    tac >out.txt <in.txt
    tac <in.txt >out.txt

They are best read as prefix operators that modify the state of the process invocation.

What's even more confusing is that you can do:

    tac 1>out.txt 0<in.txt
    tac 0<in.txt 1>out.txt

Then they look like binary operators again. But the descriptor is "stuck on", it's actually part of the operator lexically.

Valid:

    tac 1>out.txt 0< in.txt
    tac 1> out.txt 0< in.txt

Invalid:

    tac 1 >out.txt 0 <in.txt
    tac 1 > out.txt 0 < in.txt

In other words, there can't be any space between the LHS file descriptor and the > or <.

The syntax of shell is horrible, and that's something I would like to fix with Oil shell. But I don't see the problem you are pointing out.

(sleep 5 && echo hello) | > test.txt?

Workaround:

(rm file; tac > file) < file