Find|xargs like a boss (Real Example) | Hacker News Reader

Find|xargs like a boss (Real Example)

r-bloggers.com

41 pointsg-garron20 comments

Comments are closed on that page, so instead I will comment here.

There's no need to use xargs, and pipe to sh in this scenario. The same functionality can be achieved by using only find:

  PROCS=8;cuffs=`which cufflinks`;find `pwd` -name 'accepted_hits.bam' -type f -exec qsub -l ncpus=$PROCS -- $cuffs -p $PROCS -o {}-cufflinks -g genes.gtf {} \;

Creating a text stream which you then pipe to a shell is a good way of debugging complex operations, especially potentially destructive ones that may need inspection to verify. It's a technique I use a lot, to the point that some of my more destructive scripts don't do anything - they just output shell commands, and it's up to me to pipe them to the shell.

You can achieve this also by using find and its -ok option, instead of -exec, that way find will ask for confirmation before doing anything.

Thanks for the -ok option. was not aware -- but yeah, I've been a heavy user of -exec option for years, and it always blows my mind when people get crazy complicated xargs going.

That's not the same thing at all; I could emulate that in a script with 'read' and 'if', but that's not what I do.

I would have expected an article on how to use these tools "like a boss" to point out that if there is whitespace in the filenames you will run into problems unless you use "find ... -print0 | xargs -0 ...".

What I'd have expected in an article on how to use this tool is talk about `-exec` and `-exec +` before shooting straight for xargs.

I very rarely use find -exec, because I usually want to massage the list of files further. So I have a simple utility, print0, which reads lines and outputs null-terminated lines. It turns any list separated with newlines into input suited to xargs -0. File names with embedded newlines are vanishingly rare, compared to those with spaces, so it has always worked out well for me.

xargs is very useful, although nowadays I usually use GNU parallel instead (it's essentially the same but with a few advantages: https://www.gnu.org/software/parallel/man.html#differences_b...)

xargs -P is sort of similar

The link in the post to which you replied covers that; it's the section of the man page which deals specifically with differences between xargs and parallel. Having used neither of these tools before, I think two relevant lines from that link would be:

>xargs can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel.

>xargs has no support for keeping the order of the output, therefore if running jobs in parallel using xargs the output of the second job cannot be postponed till the first job is done.

Came here to say this. GNU Parallel is great.

The author would benefit from replacing his backticks with $(pwd) instead. You can nest this form arbitrarily deep as well as having it interpolate in strings or heredocs.

$(foo) is a bashism; unsupported by different (older) versions of bash and especially sh

I don't know how well $(foo) is supported outwith bash in practice, but it is POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

For the record, these examples assume GNU xargs, so they won't necessarily work with the BSD xargs that comes with OS X.

This was one I used to backup all websites in a folder:

  find . -mindepth 1 -maxdepth 1 -type d | awk 'BEGIN {FS="./"}; {print $2}' | xargs -d '\n' tar czf backup1.tar

Note that you'll have to change $2 to the correct column number for the name field depending on your output.

Why not just cp the parent directory?

This command doesn't include the files in the parent directory.

An easier way to accomplish the original command uses globbing (like in bash):

    tar czf backup1.tar */

I got that part, but why are there files there in the first place...?

Reply on news.ycombinator.com