Parallel Shells with Xargs | Hacker News Reader

Parallel Shells with Xargs

linuxjournal.com

61 pointsoedmarap22 comments

You can actually do this with GNU Paral…oh, wait, I forgot that this isn't Stack Overflow and I'm not Ole. But seriously, xargs is pretty powerful by itself; I've seen a number of awkward pipelines using heavyweight tools when xargs would have sufficed.

The article does mention GNU Parallel, but focuses on xargs.

I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator/operation kind of goes in the middle) and xargs is postfix (the op goes on the end).

  for file in *pdf *azw3; do calibre-convert $file $file.epub; done

  find . -name \*pdf -o -name \*azw3 -print0 | xargs -I {} calibre-convert {} {}.epub

Something about the former is more natural for my brain to produce. If I want to parallelize, for shell loops my muscle memory very quickly converts the above to

  for file in *pdf *azw3; do calibre-convert $file $file.epub & done; wait

... but if I have a million files my machine is going to have a bad day, whereas the xargs version is easy to limit to N parallel invocations.

I'd really like a shell that lets me define my own sugared versions of shell grammar. I imagine writing a 'forp' that would let me do

  forp $(nproc) file in ...; do ...; done

and the shell would run up to $(nproc) invocations in parallel. Anybody know of such a shell? (Don't say eshell.)

forgot to mention that if the "operation" you're doing has any sort of conditions or branching, now you're writing

  ... | xargs -I{} bash -c "if [ $(stat --printf %Y {}) -gt 1234567890 ]; then do_something; else do_something_else; fi"

whereas if you're building your command with just a shell loop, when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want.

I bet one could coerce emacs into doing something reasonable for syntax highlighting inside a quoted `xargs sh -c "..."` but ugh

Again, here we can rely on zsh having advanced globbing functionality. Instead of moving your test in to the execution block and suddenly having possibly unbalanced parallel groups you switch to something like *(.mw-1) to operate on files(the .) modified in the past week(the m[odified]w[eek]-1) or whatever. I'm choosing mtime because of your %Y, but zsh offers many other filters too. This is detailed in the glob qualifiers section of the zshexpn(1).

I'm not sure whether I'd move the actual branching in because of the balancing issue, that is point where my hairs stand up. Perhaps, this is where it all falls apart.

> when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want.

What?!? How have I never heard about this command? Thank you so much for telling me about it! This will be so much better than my current practice of manually copying unwieldy shell commands into a temporary file and running them there.

https://en.wikipedia.org/wiki/Fc_(Unix)

One more fun thing in this vein: edit-and-execute-command is a readline binding commonly / by default bound to ^X^E (at least in "emacs mode").

It takes whatever command you're working on at the shell prompt and opens it with your $EDITOR (or $VISUAL if you have that defined separately for some reason). When you save and quit your editor, the shell runs the edited command.

And here we see an additional cost of the xargs approach: the $(stat ...) in the above will get expanded inside the ""s before it gets fed to xargs, and you would have to debug that (and ultimately either use single quotes or use \$).

Nevertheless, I find xargs useful enough that I bite this bullet and have gotten very familiar with bash's quoting rules and how to work with them.

This feels like exactly the reason I prefer using zsh's zargs command for these things. In part because you can use all the extended globbing functionality zsh provides. Something like the following [untested] rework of your pipeline:

    zargs -P $(nproc) -i ***(#i).(pdf|azw3) -- calibre-convert \{\} \{\}.epub

We're firing up nproc tasks, matching case-insensitive .pdf or and .azw3 at any depth, and then batch processing them. zargs is documented in zshcontrib(1).

It doesn't feel that different to your imagined syntax to me, quite whether that matches your mental model better is for you to decide though.

More often than not, you'll get a better result (often more safely) by writing a small thing which generates the complete command lines for all the tasks which need running. Then once you are happy the right thing will be done, feed the list of commands to GNU parallel or something else which does the same thing.

Not only does this approach provide confidence before pulling the trigger, but the list of commands may be retained forever for reference / example. It's also clearer and doesn't get the reader mystified by "shell wizardry", as some people are wont to be.

The phrase "POSIX.2" is archaic. If POSIX is IEEE Std 1003, then "POSIX.1" is IEEE Std 1003.1, "POSIX.2" is IEEE Std 1003.2, and so on.

In the first revisions, 1003.1 specified "Core Services" (mostly the C language stuff), and 1003.2 specified "Shell and Utilities". But when the Austin Group formed in 1997, they decided to roll the shell and utilities in to 1003.1. There no longer is a .2; it's been deprecated by 1003.1-2001 and all revisions since then.

If POSIX.2 is archaic, then would you call POSIX job control precambrian?

This reminds me of the stagnation of CMD.EXE, the origins of which appear to lie with OS/2.

Sure POSIX job control is old and janky, but it's still in modern POSIX.1, as it has been for 20 years now; while POSIX.2 has been dead for 20 years now.

The userland behavior of POSIX is separate from the C API.

While I haven't read the latest standards, the only way to specifically address the userland is with "POSIX.2" (which is how the wiki does it).

If there is a better way to say it, then the wiki should be updated.

Also, "The New Kornshell" (by Korn) has the following commentary in the "About this book" section:

"ksh is intended to conform to the Shell Language Standard developed by the IEEE POSIX 1003.2 Shell and Utilities Language Committee."

What term better describes the purview of that committee (alone) than POSIX.2?

"The New Kornshell" (2nd ed) was published in 1995, when POSIX.2 was a thing, and there was a separate "Shell and Utilities Language Committee".

In 1997, that committee (the 1003.2 committee) merged with several other committees (including the 1003.1 committee) to form a unified committee called the "Austin Group". The Austin Group publishes a their work as a combined 1003.1; there is no longer a separate 1003.2 standard or committee.

There are still several other separate "1003.x" standards+committees for languages other than C and Shell that did not become part of the Austin Group (such as 1003.5 for the POSIX Ada interface). But 1003.2 is no longer one of those separate standards+committees.

If Korn were to write that sentence today, he might instead write "ksh is intended to conform to the Shell Language Standard developed by the Austin Group and published as IEEE POSIX 1003.1 Volume 3."

Let me ask you another question.

If I was to write another article, "Breaking the chains of POSIX," where do you think I should focus?

POSIX job control was a soft target, and I had imagined my next focus to be the "local" keyword in shell functions.

Do you have suggestion on a better weakness in POSIX to encourage improvement?

Microsoft's powershell is evolving, and the POSIX shell is ossified. I think the POSIX shell standard must evolve to remain competitive.

If you agree, where should this improvement be encouraged?

> The userland behavior of POSIX is separate from the C API.

Yes, the C API is specified in IEEE 1003.1 volume 2 ("System Interfaces", abbreviated as "XSH"), and since 2001 the userland behavior is specified in IEEE 1003.1 volume 3 ("Shell and Utilities", abbreviated as "XCU").

(Volume 1 is "Base Definitions"/"XBD" and volume 4 is "Rationale"/"XRAT").

> the only way to specifically address the userland is with "POSIX.2"

The way to specifically address the shell and utilities is to refer to the volume within POSIX.1; you can do this by number ("Volume 3"), by name ("the Shell and Utilities volume"), or by abbreviation ("XCU").

> If there is a better way to say it, then the wiki should be updated.

Yes it should. Which wiki are you referring to?

This is the wiki section that isolates the shell & utilities best:

https://en.wikipedia.org/wiki/POSIX#Parts_before_1997

In all the others, it is combined with the C API, and other OS standards.

Yes, that section of the article is discussing historical POSIX prior to 1997. The next section discusses the forming of the Austin Group and from then on only talks about 1003.1. The only place that article uses the phrase "POSIX.2" is to refer to the historical 1003.2-1992 (that "1992" is a year, if that's not clear).

CMD was based upon COMMAND, which in turn has roots in CCP.

Kind of embarrassing, but I never actually learned bash well enough to use it even after a decade of programming. When there was only one pair of feet in the sand, that’s where xargs carried me. Everything from web scrapers, to kubernetes admin, I’ve /only/ used xargs to do all of the variable splicing and parallelism that I need.

Reply on news.ycombinator.com