Why "ls *.c" is at least dumb, and possibly dangerous | Hacker News Reader

Why "ls *.c" is at least dumb, and possibly dangerous

groups.google.com

28 pointsr11t41 comments

Wait. What?

1) Is he implying that launching a perl interpreter is somehow a more lightweight operation than launching ls and having it stat() half a dozen files? Having ls check if a file exists 99% of the time won't even touch the disk. If you're working in that directory, those inodes will probably be cached. Running stat() on a cached inode is hardly more expensive than a system call. Read: cheap. Now perl launching will probably load a few libs and check a few files before even getting to his one-liner.

2) Am I the only one, who in my ~10 years of running unix-like operating systems have never created a filename with a newline char in it? Like even when opening obscure filesystems, never.

3) Does he think unlink() skips the check for the file's existence?

4) If he wanted to overcomplicate things and make sure everything's kosher, why not:

  find . -maxdepth 1 -type f -name '*.c' -delete

Whenever I see anything piped to a perl one liner, 4 times out of 5 it's some sort of hack that can be done with standard utils.

It also grinds my gears when people take longer to think of making something more efficient than the amount of CPU time they'll save in their lives of using that 'more efficient' solution. So he spent a minute coming up with this and ten minutes posting about it. Now, will he save roughly 11 minutes of execution time in his life while running this "superior" command (also count that it takes longer to type)? If not, he wasted his time.

From his sig: "Smalltalk/Perl/Unix consulting". When you have a large-enough hammer....

1) stat() is a system call, so one system call is bound to be faster than two. 2) Sneaky users do this; they create files like /dev/shm/...\n-rf\n.. and you'd better know how to delete it correctly. 3) ??? 4) -delete is a GNU extension.

1) Just the system call overhead is not much compared to what the system call actually does, so it depends on what you're calling. Two system calls versus one with identically-cached information, sure it's slower. I'm just saying perl will also probably stat() and if not perl, the unlink() system call would perform the same function. When you're modifying files and flushing it to a drive, especially while using the shell interactively, the sys call overhead is completely negligible. All I'm trying to say is claiming the first command as inefficient is groundless; you execute it as a human and once -- it won't feel any slower.

2) If cleaning up after malicious users, sure why not. Doing this in your source tree or during normal usage? I don't know.

3) He mentioned ls stat()ing files as an inefficiency since the shell does it and ls does as well.

  FIND(1)                   BSD General Commands Manual                  FIND(1)

  NAME
       find -- walk a file hierarchy

...

       -delete
             Delete found files and/or directories.  Always returns true.
             This executes from the current working directory as find recurses

4) It's still an extension: http://www.opengroup.org/onlinepubs/009695399/utilities/find...

So what ? We're not in 1985. It's almost universally available, and perl too is not part of posix.

This is not the point of what he is saying.

Read the context of the thread. It is a reply to this:

http://groups.google.com/group/comp.unix.shell/msg/30e16d21d...

Randal is pointing out that piping the output of ls to the perl interpreter is dangerous and shows the safer way to do what the original poster wanted.

You aren't going to create a file with a newline in the name. The person who wants to totally screw you over is going to create the file with the newline in the name.

I realize that. The example he used was a collection of c files, leading me to believe he was showing a personal collection of source as an example, wrongly or rightly. And I realize you aren't going to be creating files with newlines in them (or leading dashes, or many dots, etc), but once you get into a territory of someone being able to do that, and you're not competing in a CTF, I'd say there are larger issues to deal with.

I can do no better than quote Randal himself again, from the same thread:

"Well, there was a very embarassing error for Apple a few years ago for the 10.3 to 10.4 upgrade as I recall. Apparently, if you had a space in your home disk name, all sorts of files got deleted.

So, the moral is, no matter how rare you think something is, that's NO EXCUSE for not doing the right thing for all possible characters."

http://groups.google.com/group/comp.unix.shell/msg/504a75cc7... http://groups.google.com/group/comp.unix.shell/msg/0385fc2d1...

Spaces in names are not rare - space is part of natural language, a filename or volume name in considered to be a single line label.

"ls *.c" is not dangerous. The pipe is dangerous.

The user making files with newlines is dangerous.

The user doing something that the system allows him to do is dangerous??

You guys are insane.

so, rm -rf / is not dangerous?

I completely agree that the system shouldn't allow him to do this either. Both the graphical and command shells for Linux (GNOME and bash respectively) do not allow files to be created with newlines characters as far as I know.

Filenames can contain newlines?

You learn something new (and vaguley horrifying) every day.

Filenames can contain plenty of happy stuff.

$> touch -- --

$> rm --

usage: rm [-f | -i] [-dPRrvW] file ... unlink file

$> rm -- --

$> touch \

$> touch \\ \ \ . \-

favorites of malware authors everywhere.

I instinctively cringe whenever I see an odd looking 'rm' command.

On a sidenote, creating a -i file in important directories can help against accidental "rm -rf"s. The -i will be given to rm as switch and rm will now prompt for each file. Of course backups are better but this might just safe your ass that certain time.

Traditionally Unix filenames can contain everything except for / and null (aka \0). This is why the find utility provides a facility for printing null terminated filenames (-print0), when used in conjunction with xargs -0 this is a fool proof way of iterating over files in the shell.

For example, to delete all .o files in the current directory and its subdirectories:

   find . -name "*.o" -print0 | xargs -0 rm -f

Why not just

  find . -name "*.o" -delete

?

That's fine for /this/ example because find provides that as an option. But you might want to grep for something in many files of varying depths, or run some home-brewed script on them for which find does not provide a built-in. -exec works of course, but spawns grep (or whatever) once per file; using xargs saves considerable time since it passes lots of files to 1 command.

Or, you can use zsh.

    rm **/*.o

    grep foo **/*.{java,sql,xml}

    # Where is that blasted IBM utility?
    ls **/*pmt*(x.)  # list e(x)ecutable files (.) that have "pmt" in their name.


    # Show me all the "Setup" dirs in my project
    ls -d **/Setup(/)

Yep. Though I don't think I've ever seen an actual legitimate use of this capability. I mean really, what's the point of an embedded newline...

Actually the Mac uses (or at least "used") a hidden files named "Icon\r" (yes, an embedded carriage return) for its custom folder icons. I'm still not sure that counts as "legitimate".

I had to write a one-liner to deal with that on Mac OS 10.4 -- had to detect if a user had placed anything on their Desktop, which had a custom folder icon because it was actually a cross-filesystem symlink:

  desktop=$(test -z $(ls -A1 /Volumes/Documents/Desktop | \ 
            sed -e '/^.DS_Store$/d' -e '/^Icon^M$/d'))

The ^M was a literal CR, which would get munged by all my usual editors except vim. I'm sure there's some sturdier way to escape it...

I remember choosing to use ls intentionally, but not what sucked about the alternatives.

As noted above, I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters, and I approve of them.

This article does point out one thing that I've always found really strange/dumb about Unix. Why does the shell expand the wildcards? That's not how it works in Windows, for example.

I periodically have to cleanup abandoned and overflowing unix mailboxes where every message is it's own file. You'd think a simple "rm *" would work, but it doesn't, because you get an error message saying that there are "too many arguments". From my perspective there's one argument, the asterisk. Very confusing. I end up having to use xargs to get it work. Seems like a poor design.

It's so that every script in the world doesn't have to duplicate the logic for how to interpret globs. They inevitably won't end up doing it exactly the same way.

As a Windows developer, I can tell you with certainty that practically no one does it even close to the same.

Shouldn't that be part of the OS API? Why call me with 1,000 arguments, when I could call an API with the wildcard and get the 1,000 files back? My poor argv can't handle it the other way.

this is one of those things no one cares about until it affects them.

> Suppose you have a file with a name of "aaa\nbbb.c". (from the article)

To be honest, I wouldn't care about my shell not handling newlines in files, I'd care about my users creating files with newlines in them. Space is natural language, newline is not.

I wonder how many files with a name containing \n are in the wild. And if you encounter such a file, does it really matter if you accidentally delete it?

Waisting brain cycles to save CPU cycles (in the wrong places) is shameful, insulting and boring.

It would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)

I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters. I approve of them.

With regard to why this matters, it's not uncommon for badly written scripts to interpolate filenames directly into commands. Consider this Perl fragment:

  $sessionFiles = `ls /tmp/session_*`;
  system("rm -f $sessionFiles");

and what it would do with a file called:

  "/tmp/session_blah; chmod +s"

And if you encounter such a file, does it really matter if you accidentally delete it?

It matters. Leaving a bug around is disgusting and wasteful especially when you can fix it.

* would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)*

Fixing bugs and writing programs that have as few bugs as possible is very important.

When you find a bug, by all means, go ahead and fix it. (Well, not always. You need to make sure your fix is not likely to introduce a more severe bug. Not all bugs made equal and some guts-feeling required).

If your goal is to make your system as robust as engineerically possible, you should most likely pay more attention in some directions and less in other. It's not only about quantity - you better have no critical bugs then have only-a-very-few critical ones.

The problem is the file you accidentally delete (or whatever) is probably not the one with the weird name. Most bogus filenames I've seen were created by shell scripts that had a set of commands that was accidentally quoted instead of executed, so they tend to contain the names of files which were important to the author.

Am I the only one who thinks that scripting is a "Dirty Job"?

I have not written nor seen any bash script that is neat, clean and fairly clear. It all seems like a very shady hack that someone think is smart.

Not sure why you're being downmodded. On the simplest of engineering levels, shells don't adequately separate data from execution. They also lack data structures beyond strings and arrays.

ipython / iruby can and do make much nicer shells out of the box.

Dumb on whose behalf? If you need to know the inner workings of the shell in order to know that is "dumb" then I submit that in and of itself is dumb.

Apparently you have not been bitten by a filename expansion bug when shell scripting. This kind of thing is fairly common with bash and some of the other ones when dealing with files that have spaces/newlines/whatever. It boils down to learning and knowing your tools and how to work around their limitations.

In this case, if you're not an expert with bash, a good rule of thumb would be to pipe your glob-expanded list of junk into perl -nle 'print' (or echo, or... etc) before running something destructive.

Every time. Until you're an expert.

Reply on news.ycombinator.com