A small primer on xargs | Hacker News Reader

A small primer on xargs

bitops.io

48 pointsbitops19 comments

"In particular, the presence of \0 in your data will throw xargs for a loop (no pun intended). In this case, it is necessary to use the -0 option. This is particularly relevant if you are using the find command and supplying the -print0 option."

This glosses over the fact that those two things are intended to be used together. Delimiting filenames with a \0 is superior to delimiting them with spaces or newlines, as filenames cannot usually contain ASCII NUL; xargs and find can therefore be used to remove heterogeneous lists of files.

Additionally, the example given may better be worded as "find . -name '* junk* -exec rm '{}' \;" or even "find . -name '* junk* | xargs rm". (as given it strikes me as a useless use of grep and is additionally fragile if ls is aliased to "ls --color=always", spewing ANSI into

EDIT: Please excuse the additional spaces in the find command arguments, they are a formatting avoidance feature.

Little known fact: the -0/-print0 idea is due to DJB.

If we're ripping this sample apart, I'd like to point out that

  rm *junk*

in this case would've done the same, without spawning lots of useless stuff.

xargs is cool. The example fails to show a decent use case. For me a line like that would indicate that someone is _not_ familiar with his tools. Similar to

  cat somefile | grep something

instead of

  grep something somefile

Pipes and processes aren't endangered or (in practice) limited, but we still don't need to be wasteful?

Furthermore, parsing the output of `ls` is a terrible practice[1] that makes for an even poorer sample.

A little further in the article, the author even suggests running

    `ls | grep junk`

(instead of the much simpler `ls junk*`. Do I need to point that out?).

I also know some people are going to retort that wasting a process or a pipe is not that bad. This discussion has been had over and over again, probably ever since Unix was invented.

In any case, the poor example chosen, as well as the flaky commands used, seriously hinder the credibility of the article.

[1]: http://mywiki.wooledge.org/ParsingLs

They don't do the quite the same thing.

   ls | grep junk

is more similar to

   ls -d *junk*

You can argue the first is simpler. You can also argue more successfully that it is more general.

edit: I acknowledge the points in your link. Advocates for the unix "bunch of small tools" philosophy tend to overlook its weak spots.

Instead of -exec rm {} \; you could use -delete option as well.

This option is not portable at all. For instance, it won't work on Solaris' find. I strongly advise against the use of non portable options.

It may be a good idea when writing scripts that have to be portable.

But there's quite a lot of nice non-standard options in the GNU userspace tools, and it would be a shame not to use them when available.

If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces.

Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.

> more likely is its aliased to "ls --color=auto" or similar which will work normally.

My ls (which I haven't much reconfigured) behaves very differently depending on whether it's outputting to a shell or a pipe or redirect. Not just colours, but columning and filetype indicators as well.

"What xargs is doing is taking each line of input and applying the supplied argument or command to the line of input"

Actually it does not run command once for each line of input. This confused me for some time until I read the man page:

"The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option"

Think you're looking for "-n 1" if you want that behavior..

I was going to respond on this very point myself. This blog entry is just plain wrong wrt the way that xargs works.

xargs is so 20th-century. parallel[0] is way more awesome (especially with all the multi-core systems everybody has now). Cool bonus: all the things you learn for xargs work in parallel too!

[0]: http://www.gnu.org/software/parallel/

I Was wondering what the difference with running parallel jobs using xargs -P was, but that's covered in their man-page: http://www.gnu.org/software/parallel/man.html#differences_be...

A billion times, this. I just learned of GNU Parallel about 3 months ago and wish everyone knew about it. Have seen it used to stunning effect with a comically large pipeline (OCR patents) across a comically large number of EC2 instances. Very awesome indeed.

I try to use perl instead of sed, awk, bash each time the needs are beyond basic usage. This applies also sometimes for xargs for example with the following command. find . -name junk\* | perl -nle unlink

My favorite usage of xargs if for parallel operations: cat hostlist | xargs -P 0 -n 1 -I host rsh host gzip -9 log/*.log

ls | grep junk | xargs rm # is dangerous due to The separator problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem.

ls | grep junk | xargs trash-put

https://github.com/andreafrancia/trash-cli/#readme

OP here - thanks all for the comments - so yes, it looks like I don't know xargs as well as I thought I did. I'll update the post to be more accurate.

As for the points about a bad example, I am fully aware that it is extremely contrived. The point was to not confuse anyone with commands they don't already know. I'm wary of posting examples that necessitate people learning 4-5 new commands when their focus is learning just one.

Reply on news.ycombinator.com