Hacker News Reader

Hacker News

Comment by sabat | Hacker News Reader

Parent Full thread

sabat

Recursively find all Emacs backup files and remove them find . -name '~' | xargs rm*

Recursively find all Python files and search them for the word ‘import’ find . -name '.py' | xargs grep 'import'*

Hmm. I don't mean to be a tweak, but you don't need xargs to do either of those things. Just:

find . -name '~' -delete

find . -name '.py' | grep 'import'

Note: I can't figure out how to get an asterisk to show up, and don't have time to look it up.

View on HN

The -delete option is not available on all UNIX systems, as it's not part of the POSIX spec (http://pubs.opengroup.org/onlinepubs/009604599/utilities/fin...).

(Also, rant rant, I really don't understand why find was extended with -delete in the first place. What's next, "ls --delete" or maybe "cat --grep"?)

The orthogonality ship has sailed with Unix, for better or (IMHO) for worse. The Unix Haters were right about this, among other things, and the sections of the handbook about the shell are still valid, even as time and Moore have rendered others quaint.

For worse, I concur. I still fire up my SGI boxes once in a while to remind myself things didn't use to/don't have to be as binary as they mostly are today (Linux vs BSD, iOS vs Android, Intel vs AMD etc.). That, and to run electropaint, of course. =)

-delete was added to find because of the race condition with doing it using xargs.

See section 9.1.5 http://www.gnu.org/software/findutils/manual/html_node/find_...

Thanks for that link.

It walks through many of the same issues as the OP, but with more sophistication.

It also explains "+", which is used in place of the traditional ";" to essentially get xargs type argument accumulation, but within find.

That is,

  find . -name '*~' -exec rm {} \+

I did not know about that. It's in Mac OS 10.6 find, for one.

That was already fixed with -print0 | xargs -0, but then this solution is dismissed with "The problem is that this is not a portable construct;...". The -delete isn't either, so this is a straw man argument, although it probably is the most efficient and secure of all.

Read it again.

-print-0 | xargs -0 does not fix the race condition.

The problem is someone can swap in a symlink after the find, and before the xargs.

I'll have to run some more tests, but I can't see how -delete would help in that case .

Because find changes to the directory first (carefully not following symlinks), and then deletes the file from there.

It does not delete the file using the entire path (which may contain a sudden symlink).

It's not possible to do this safely using xargs.

Take a look also at -execdir which does the same thing - changes to the directory first, and runs things from there. -exec is not safe and should not be used.

xargs is not safe if you are running against a directory not your own. You should use find and -execdir instead.

Yes, the original authors of posix made a mistake here.

> Also, rant rant, I really don't understand why find was extended with -delete in the first place.

I'm hoping you understand it now.

Yes, thanks for the extensive info.

Rob Pike and Brian Kernighan warned about this trend in their seminal paper "Program Design in the UNIX Environment" (also known as "cat -v considered harmful" which describes how proper unix programs should be designed:

http://harmful.cat-v.org/cat-v/

I did a quick look online to see who has it. GNU, DragonFly BSD, NetBSD, and FreeBSD all have a -delete option. OpenBSD seems to be the only one that doesn't (although its online manpage does give examples on how to do it with -exec and piping to xargs).

That's basically two flavours of UNIX (Linux and BSD), but that's OK - others are unfortunately either dead or dying. The UNIX family tree (http://www.levenez.com/unix) has been shrinking so rapidly in the last few years, that the current situation looks like the early years.

Since the BSDs are all independently developed, I don't consider them to be one Unix, even though they share ancestry.

I'm not an expert by any means, so please provide correction/feedback if I'm wrong, but I've read that xargs is more efficient because it can send parameters to the cmd in batches. I've read this in contrast to `-exec {}` not `-delete`, however.

For example:

    find . -name '*~' -exec 'rm {}'

The statement above executes `rm result` for every result. By contrast:

    find . -name '*~' | xargs rm

The example above would group the results and pass them to rm like this:

    rm result1 result2 result3 result4

Because I'm not an expert, I don't know how many parmeters it will pass, or if initiation of many new processes has a significant performance impact on newer systems. I would suspect that for anything involving the disk, I/O will be the bottle neck, not process turn-up time.

Anyone have an opinion/insight?

-delete will be the fastest because find can do everything, and it already has the file loaded, and doesn't have to do a second lookup.

-exec will be slower because for each file it has to spawn a new process (this is where most of the time goes), and then that new process has to look up the file.

xargs will be faster than -exec because it will collect a few hundred, or thousand, filenames, and pass them all to one command (the article claims 4096 is the default, on some systems it may be lower). This means that typically, only 3 processes need to spawn: `find', `xargs', and one `rm'; instead of find, and many, many `rm's.

Now, xargs is still slower than -delete because it will buffer the filenames, either waiting for the list to end, or 4000 filenames to pass to `rm'. Then, rm must look up the file from the filename.

To make my point I just set up a test situation with 1110 files ending with `~', among a total of 2221 files. I tested how fast it is to delete all the files ending with `~' using the 3 following commands:

   $ find . -name '*~' -delete
   $ find . -name '*~' -exec rm {} \;
   $ find . -name '*~' -print0 | xargs -0 rm

         | real   | user   | sys
 -delete | 0.024s | 0.003s | 0.020s
 -exec   | 0.819s | 0.007s | 0.107s
 xargs   | 0.073s | 0.007s | 0.017s

I probably should have run those tests multiple times, and taken the average, but meh.

Very cool. Thanks for the insight.

Mind if I ask how you timed execution?

The take aways for me are:

* Use find's built-in options where possible

* Use xargs where a built-in method isn't available

* Use -exec only when nothing else will do

I timed it with the `time' command :P

Most systems have a find command that supports the "-exec command {} +" option, which does not spawn the command for each result, but instead puts as many arguments as the shell will allow on the command line.

Nobody remembers the various command line switches of find or grep. I use xargs quite often because it's a very simple concept and I know how to mix and merge any commands together with it and that's something I've only had to to go ahead and learn once.

The grep one didn't involve a switch on grep. It was just the xargs example with xargs removed.

The grep one is also wrong as it does an entirely different thing without xargs.

No. You're looking for files having import in their path here, not in their contents.

He got the command wrong but his point stands:

    find . -name '.py' -exec grep -H import '{}' \;

No he doesn't. That's a completely stupid compared to xargs as soon as you have more than a few files. You're starting one grep process per file.

Thanks for the laugh though.

All of this is stupid compared to just using a good shell and doing grep import * * /*.py.

find . -name '.py' | grep 'import'

I don't think that works. That'll grep for files named '\import\',

I use xargs for the find + rm cases he mentions. However, if you just want the file names containing the words import, I rely exclusively on grep:

grep -r -l import *.py

You are correct about -delete. You could also use -exec to execute arbitrary commands for each file returned by find. I needed some examples however, and I often see "find | xargs" because people don't know, or forget, the options to find. I should write a follow up post on find.

In addition to ptramo, you really want:

grep 'import' `find -name '* .py'`

Well, you want xargs as soon as your project is big, because you'll reach the limits on arguments fairly quickly.

I won't go too deep on the issues this will trigger if your filenames contain special characters. We often think of spaces, but you could have some nastier stuff. If you want to sound smart at your next geeks reunion, simply read http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h...

On (modern, i.e. 2.6.23 and later) Linux at least, you won't hit them quickly:

    $ getconf ARG_MAX
    2097152

and if you do hit them, they're very easy to raise:

    $ ulimit -s 32768
    $ getconf ARG_MAX
    8388608

xargs is still useful for its other features, of course.

# echo "2097152/$(echo "/my/still/quite/reasonable/path/to/a/file"|wc -c)"|bc 49932

linux-2.6 # git ls-files|wc -l 36747

Getting close! :)

And I would guesstimate (Linux, kernel >= 2.6.23) to still be a fairly small amount of the machines people interact with professionally through a command line.

And if in a script/snippet, you often want to cover a vast majority of the systems you _could_ end up with. Won't be System III, at least for me, but there has to be a RHEL5 system in a closet right? :)

That's the kernel's limit, Bash (3) has its own limit, which is significantly lower, I've hit it before. I think Bash 4 can do an arbitrary number of arguments, within the kernel's limit.

No, bash doesn't and didn't have its own limit. On an ancient system (Debian Sarge), ARG_MAX=131072:

    $ bash --version
    GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
    Copyright (C) 2002 Free Software Foundation, Inc.
    $ strace bash -c '/bin/echo `seq 1 30000`' 2>&1 | grep exec
    execve("/bin/bash", ["bash", "-c", "/bin/echo `seq 1 30000`"], …) = 0
    execve("/bin/echo", ["/bin/echo", "1", …) = -1 E2BIG (Argument list too long)

As you can see, the argument list too long error came back from the execve syscall, i.e., from the kernel. (Note that I shortened the strace output to make it fit the page)

Of course, I meant my version as an alternative to xargs. And I think you have about 128k of space for the filenames, but yeah with large projects that can be a problem.

Thanks for the link, that's more interesting than the submission. :)

Fair point, I just submitted it :)

Or maybe, if you have GNU grep:

    $ grep -R --include=\*.py import .

Reply on news.ycombinator.com