Fixing Unix Filenames: Control Characters, Leading Dashes, and Other Problems

Bourne shell is notorious for its problematic quoting, both of filesystem data and of any data from any other source. Every example in which he described a problem with a filename parameter could just as well be a problem with a non-filename parameter. The correct solution is to not program complicated scripts in Bourne Shell, and instead use a language which does not implement variable access by interpolating strings and then re-tokenizing and re-evaluating them. Examples of satisfactory languages include Perl, Python, Ruby.

Regarding UNIX arguments and the dash, it is an unfortunate aspect to the flag argc/argv/envp calling convention for unix programs. Some other operating systems provide more structure in their calling convention, explicitly separating different types of parameters from one another. This is both a strength and a weakness, as it results in a uniform yet inflexible systems interface. One of the greatest strengths of UNIX is that its calling convention is so flexible. The semantics used today are quite different from the semantics used 40 years ago -- yet execve() remains unchanged. I would encourage anyone interested to do a bit of historical digging here, and see how those more ridged system APIs fared over time.

Anyway, the solution to his initial question of using `ls` is the -- argument, which signifies argument parsing should be disabled for the remainder of argv: ls -- *

The correct answer to his dotfile/glob question is: "glob() and the Bourne shell do not have the semantics you're after. Do not use them, use readdir()."

The correct answer to his find -print question is: Yes, print's use of whitespace was a mistake, and it is a mistake repeated continually throughout the land of shell scripting and accompanying standard UNIX utilities. As he notes, it is why print0 was introduced. Making print0 standard is far easier than reworking filesystem semantics (and, reworking userland in this manner is a more complete solution as it addresses data integrity issues from non-filesystem inputs as well). If you want reliable, correct programs, do not write them in shell.

All of these problems arise because we're passing around text that needs to be parsed. Programmers don't know the parsing rules, And they vary by utility. The end result is that it is very easy to make something work, and very hard to make it work correctly.

As was noted by fffggg, most of these problems disappear as soon as you switch to a language that lets you stick the filenames in strings and never again tries to parse them. All of the major scripting languages will do.

But not all, just most. The exception being the UTF-8 issue. It does not solve printability of strings with unknown (and perhaps no valid) encoding. Furthermore scripting languages often will want to try to interpret external bytes coming back from the filesystem as a string, and may have trouble if file names are not some recognized encoding.

    $ find $CRITERIA -print0 | xargs -0 -r -n $BATCH_SIZE $COMMAND

Usually COMMAND would have '--' at the end to prevent filenames from being parsed as arguments, but not all commands need that treatment. If COMMAND doesn't take the filename list as the last argument(s), it can be fixed with the '-i' option to xargs.

The BATCH_SIZE part here is important in many cases. Since large numbers of filenames will break the command line arguments size limit to COMMAND, a reasonable BATCH_SIZE will prevent thousands or millions of filenames from breaking your script. Also, sometimes you may want to run the files through COMMAND one at a time (BATCH_SIZE of 1).

Finally, the '-r' option, which I don't see mentioned often enough, tells xargs not to be derpy and run COMMAND with an empty argument list. Seriously, '-r' is short for '--no-run-if-empty'.

For anything else, a nice scripting language is the only way to fly.

The whole point of Unix, and the reason that it was so revolutionary, was to empower the easy composition of complex tasks via scripts and command line invocations. The “negative freedom" implemented by the filesystem has undermined the "positive freedom" of Unix's original intent.

The kernel hackers' response to this, back in the day, was typically, "Program in C instead." It was a shame that the kernel hackers didn't understand Unix!

Taking care of a directory full of madness is as easy as `rename -z`:

  -z, --sanitize
      Replaces consecutive blanks, shell meta characters, and control characters 
      in filenames with underscores.

cat * > ../collection becomes

perl -e 'map{open ($f,"<",$_);print <$f>;close($f)}<*>' > ../collection

Strangely, it is far easier to write a correct script in perl than in shell.