Useless Use of Cat Award | Hacker News Reader

Useless Use of Cat Award

partmaps.org

33 pointsshawndumas55 comments

Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT. Therefore, to get the lines from a file into STDIN, I usually cat it. It's pretty common for me to write something like:

    cat file | grep xyz

Of course, this is completely unnecessary and a "waste" of using cat, as grep accepts as input a file or list of files to operate on, as do many other utilities. However, I don't care. It's additional mental clutter to have to remember the way to specify input from a file in addition to stdin when I know I can always cat the file and pipe it to whatever command I want.

I'm going to agree with you here. 99% of the time this UUOC stuff is just premature optimization. If it gets the job done it does not matter if it wastes a few milliseconds.

The problem is when I see UUOC use in fairly large scripts. I've seen load averages shoot through the roof due to numerous separate commands that are being run causing needless context switching as each set of commands is being run per line of a large datafile.

Fixing it at that point is no longer a "premature optimization"...

Wait, what? You mean like this?:

    cat somefile |while read line
    do
      echo $line|cat -|pipe...
    done

Because otherwise I can't see how a pipeline would be started for each line of a datafile ?

It's more common to see this issues with something like this:

   for f in *;
   do
      cat $f | ...
   done

Now make that a directory with thousands of files or a script digs down through through the tree.

But it's not really the "cat" that's the problem here, is it?

Sure it's possible without running a subprocess, just open the file on some other filedescriptor.

    exec 5<words.txt
    while read line <&5 ; do
        echo "line read: $line"
    done
    exec <&5-

(1) Like dpcx said, dealing with that isn't premature optimization.

(2) Be careful about load averages, lots of people don't understand them. It just means the number of processes ready to run over a period of time. It is entirely possible to have a very high load average with a very low cpu utilization. A bunch of little processes that hop on and off the run queue really quickly because they only have a little bit of work to do at a time can kick the load average up way higher than a single self-contained process doing the same amount of work with the same amount of total cpu utilization. The single process wins on context switches, but modern cpus cache those in hardware so its barely a win any more.

Exactly. Things like the "useless use of wc -l" fall into a similar category for me. When I want to count the lines of stdout, I use "wc -l". It may not be the simplest, most efficient or fewest keystrokes, but it is extremely easy to remember and fits well into the model of a pipeline of simple single-purpose tools. I tend to use cat at the beginning of pipelines just because it keeps the flow left-to-right, and makes it much clearer what the source of the data is.

There's value in knowing some of the advanced things that the textutils and bash can do together, and sometimes there's value in using those things. I don't dispute that, but I do dispute the need to shame people who use the tools in ways that you personally don't like.

Shame or not, I'm going to keep using cat at the beginning of my pipelines unless I have a really good reason not to. If that makes some people think I'm not in the Unix wizard in-group, I don't really care.

Pipes also match my thought flow. "wc -l" is a good example.

"Ok, I want to read this file..."

    cat file

"Then I want to find foo"

    cat file | grep foo

"Oh, and I want to know how many lines of foo there are"

    cat file | grep foo | wc -l

In this trivial example, I could have inserted "-c" with only a few keystrokes, but on a 200 character string of commands it's faster and easier to tack "wc -l" on at the end. Plus I'm not strong enough with grep to do every manipulation I might want, so sometimes that grep will be followed by some perl, or a sort, etc.

FWIW, that's the exact same way I approach that sort of manipulation. Start with the most basic operation, then construct a pipeline going left to right, adding filters / operations as needed to get to the result.

If you're doing a one-off pipeline on the command line, you can even start with less!

"I want to look through this file..."

  less file

"Oh, that's long, I want to look at all the foos..."

  less file | grep foo

"How many foos are there?"

  less file | grep foo | wc -l

Try it, it works!

This is not some magic behavior of pipelines. Rather, less (like many other terminal programs) is explicitly programmed to behave differently if the output is a terminal. Specifically, the less source code has this comment in it:

"/*

* Output is not a tty.

* Just copy the input file(s) to output.

*/"

grep does this to, which is why the color-coding control characters don't mess up other commands.

As an alternative, you can always rewrite:

   cat file | grep xyz

To:

    <file grep xyz

No? Maybe you meant "grep xyz <file" ? In what shell does your line accomplish the same thing?

[edit: Ah, I had an extra pipe in there. "<file command args" does indeed work --- "<file | command args" does not]

   <file grep xyz

works in bash for one.

>In what shell does your line accomplish the same thing?

Any descended from or cloning sh, or csh, or rc… i.e. pretty much all of them.

Just a cute tangent: I once wrote a shell with no built-in redirection. You executed a program named "<" for that, like your second example but with a space:

    < file grep xyz

(The trouble with this design: it doesn't generalize well to pipes.)

This form is also nicely modular, setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source:

   cat file | grep xyz
   gzcat file | grep xyz
   xzcat file | grep xyz
   pv file | grep xyz
   curl url | grep xyz

There are many places I get data, and it sometimes changes (e.g. I decide to compress a file and read it back with xzcat instead of cat, leaving the rest of the script unchanged). Unless you give me an alternate syntax that can handle all of them, I'm not interested in the fact that a particular utility has chosen to provide its own idiosyncratic syntax to special-case only the 'cat' data source, with an -f parameter or whatever.

> setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source:

Just replace

    cat file |

with

    <file

It still keeps the data source on the left, and easily swapped out:

    <file grep xyz

I agree, and like the fact that you can then read your commandline as a series of steps, going from input to output.

However, in bash (and other shells?) you can write "<file grep xyz" rather than "cat file | grep xyz".

You could argue that people who complain about wasting a cat are ignoring the Unix philosophy of "do one thing and do it well." Moreso with wc -l vs grep -c. If I know that wc -l is how to count lines, I don't want to remember the options for counting lines built into every other command.

>Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT

Yeah, that is actually very much the point. You are passing the contents of the file in to grep's STDIN. So why not just do exactly that rather than cat it first for no reason? You don't need to remember how to tell commands to read from a file, you just need to learn the basics of how your shell works.

    <file grep xyz

Fuck Google for the link rot, everything that points to Deja News is now useless. When will they start remembering their own corporate motto?

My favourite seemingly-useless-but-actually-not-useless use of cat:-

    ls | cat

Why is it not actually useless?

It makes the output for ls not a terminal, so ls won't try to present the output in a nice human-readable format, with columns and colors etc.

Of course there's switches for all that in ls somewhere, but still...

Or just

    ls -1

(that's a digit one)

Ah, but with the --color=auto option on ls (which I have set in my aliases) there's a difference between "ls -1" and "ls | cat"

Indeed there's even a difference between "ls -1" and "ls -1 | cat" if "--color=auto" is specified.

I use this for programs which otherwise output to pagers, when I'd rather have the output just in terminal scrollback. (And unlike the -1 option it's generic to all programs with terminal behaviors.)

Yep, I see too many senior people violate UUOC/UUOG ie..

  cat file | grep bar | awk '{ print $1 }' | sort | uniq

which can be

  awk '/bar/ { print $1 | "sort -u"}' file

The best resource if you really want to be a better shell citizen (assuming you use Bash) overall is #bash on irc.freenode.net and Greg's bash wiki.

http://mywiki.wooledge.org/BashFAQ

Honorable mention also for #awk on irc.freenode.net also which has taught me a lot.

I would still use the former.

Chances are, I'm not going to remember awk's somewhat weird, more unfamiliar syntax -- everyone knows about cat, everyone knows about piping, those two go very well together.

Using cat with pipes typically linearizes $x operation in our minds, it makes it easier to come up with solutions, and makes it easier to change them as modification is needed.

Using cat with pipes typically linearizes $x operation in our minds, it makes it easier to come up with solutions, and makes it easier to change them as modification is needed.

Yes, definitely. Also it's easier to build up from simple to more complicated pipelines that way, testing whether each iteration does what you expect it to.

I don't think it's fair to get awk involved. It's a scripting language, not a single-purpose utility.

It's silly when set up like that. But I still prefer the separate invocations -- they become building blocks. It means I can replace the "cat file" with "echo testdata", it reads nice, and for more complicated scripts, it's easy to format across lines:

    zcat /var/log/somdeamon/* \
     | grep "simple event like maybe 'error'"
     | grep "complicated date expression or whatever"
     | some other filter
     | summation?
     | maybe sort and uniq

I can start that with a simple (z)cat of one file (rather than the thousands of lines I might want to work on) -- and it doesn't require much arcane knowledge of each of the tools used, for each step.

I do agree that awk might be a little under-appreciated, but if you really feel that way, maybe you should just use snobol.

To me a "wasted cat" clearly states intention: get/feed data. Then the rest of the steps either filter, sum, or manipulate data.

The final thing that makes the "waste"-argument silly, is that Linux[1] has pretty lightweight processes. I'm not sure what the state of the various BSDs are, but it'd surprise if at least FreeBSD hasn't gotten a lot better on this too. Either way, one would think that any text pipeline would be io-bound, not limited by the number of forks...

[1] http://bulk.fefe.de/scalability/

Your first line of code is much better. It optimizes for clarity, presenting the programmer's intention in a simple logical sequence. It also likely optimizes for programmer time and mental expenditure.

Why not:

  awk '/bar/ { print $1 }' file | sort -u

I find those kind of examples a little unfair as what you're actually doing is comparing the syntax of different shells/languages (the Bourne Shell vs awk). You could write examples in Perl or Python as well - to prove how pointless awk and sort is as well if you really wanted. In fact you could even rewrite sort in awk.

While I disagree with most of the comments on here about how:

    cat $file | grep "pattern"

is easier to read than

    grep "pattern" $file

(in my opinion, if you're a sysadmin then you should know how grep works and thus the optimised solution shouldn't be any harder to read. But at the end of the day, whatever gets the job done).

However I do draw the line at your example because you're now comparing apples to oranges and suggesting that everyone should learn an additional programming language on top of memorising every standard Unix/GNU/BSD command and their flags.

What about this?

    <$file grep "patter"

It keeps the flow of information left-to-right, but also avoids UUOC.

Are you sure about that, what if file contains the equal sign?

  $ touch foo=bar
  $ file foo=bar
  foo=bar: empty
  $ awk '/bar/ { print $1 | "sort -u"}' foo=bar
  
  
  
  
  
  
  bar
  
  screw this!
  ^D
  bar
  $ awk '/bar/ { print $1 | "sort -u"}' <foo=bar
  $

The "for f in `ls * `; do" example is actually not useless. It has the quality that the loop will not run if there are no files, because ls produces no output. Compare:

    $ for i in `ls * 2>/dev/null`
    do
        cat $i
    done
    
    $
    $ for i in *
    do
        cat $i
    done
    cat: *: No such file or directory

Imagine that instead of cat we have a large, complex, expensive loop and don't want to pass a useless '*' to it if we can avoid it.

I felt painfully ignorant when I realized this was nothing to do with felines.

Another handy bit of syntax:

  $ command-name 2>&1 | less

This combines standard out and standard error into one stream so you can read all the output in less [1]. Piping just standard out leads to stray error lines popping up on the screen while less is running, and they aren't kept in less's buffer so you lose track of them.

[1] http://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/

Shorter version in bash: |& less

It redirects both outputs at once.

Be careful when redirecting STDERR to STDOUT if yo uare also planning to redirect STDOUT to a file. The order in which things are done affects the outcome. These are not the same:

  # Redirect STDOUT to file, and STDERR to file
  command >/tmp/command.out 2>&1
  
  # Redirect STDOUT to file, and STDERR to screen (as STDOUT)
  command 2>&1 >/tmp/command.out

The important thing to remember is that 2>&1 is redirecting to what STDOUT is currently pointing to, not redirecting to STDOUT itself.

Thank you for posting that. I always knew the two behaved differently, and how - but I never really understood why. Your final line cleared that up.

Recent version of bash support the (much nicer) csh syntax:

  command >& /tmp/command.out

I wanted to mention that in the original post, but I can never remember the syntax and was rushed. :(

The fact that the same thread ("but but but, I think it's cleaner / nicer / not that much of a waste / my privelege to waste processes!") springs up virtually every time the Award is posted is also Ancient Usenet Tradition.

I will accept UUOC awards only from people who never use rsync when tar or cp will do.

Though I've read this before, I've never picked up on the fact that you can do:

> <file command

It makes sense because the first statement is really saying "set up an input pipe" and not "This points at the command to my left."

    cat file | grep string

It's a gratuitous use of cat, it's not useless.

gratuitous - uncalled for; lacking good reason; unwarranted

    grep string file

Is all you need.

Useless Use of Shell Award

When there are perfectly valid directives in your language, don't just shell out. We ruby people do this too often.

Apparently, everything you do in bash is useless unless you're playing bash golf. Nevermind the fact that the commands still output what you want.

Reply on news.ycombinator.com