Bash redirections cheat sheet | Hacker News Reader

Bash redirections cheat sheet

catonmat.net

93 pointsjohndcook32 comments

One thing worth noting is that "redirection of streams" is a poor metaphor.

If I have two streams,

  1 ---------- 1'

  2 ---------- 2'

and I redirect the 1>2

  1 ----
        \
  2 ---------- 2'

it seems to work okay.

But if you have three,

  1 ---------- 1'

  2 ---------- 2'

  3 ---------- 3'

and try "redirecting 1 to 2 then 2 to 3", the natural reading would be

  1 -
     \
  2 -----
         \
  3 ---------- 3'

But that's not what you get, because you're not "redirecting streams" - you're copying file descriptors. This is partially fixed by informing people to do the operations backwards, but there are corner cases where that breaks down, and "copying file descriptors in the order they appear on the line" never does because that's genuinely what's going on.

  1 -> 1'
  2 -> 2'
  3 -> 3'

foo 1>2 2>3

This is "run foo copying fd 2 to 1 and 3 to 2"

which gives

  1 -> 2'
  2 -> 3'
  3 -> 3'

Is there a typo: s/genuinely/not geniuinely

I guess what you're explaining is the process that happens as the kernel reserves fd's? And when you start your program there isn't necessarily any guarantee that a particular fd will be available, except for 0,1,2, right?

Copying file descriptors is genuinely what's going on, so understanding it that way always works.

Edited for clarity:

The way file descriptors work is that the kernel maintains, for each process, a table mapping from numerical file descriptors to structures describing the open file/socket/device associated with that descriptor.

When the shell is setting up to spawn a new process (or subshell, for that matter), it processes the redirections in order, left to right. For redirections of the form "A>&B" this is a call to dup2(B, A), which has the kernel copy the entry in the table at B over the entry at A.

Incidentally, you will note the direction of the symbol doesn't matter when cloning file descriptors: 5>&7 and 5<&7 mean the same thing - you're just copying a file descriptor. It does matter when opening a file, as the file will be opened for reading or writing before before being dup2'ed over the requested descriptor. Having said that, of course use the correct symbol - it's good documentation.

Thanks for the this. In my opinion, there's too little written about passing fd's (use of dup2). It seems like a technique that is more useful than just as part of shell redirection. Safer than popen()?

I'm pretty sure popen is just a wrapper around this stuff. It's basically,

pipe: get two fds connected by a pipe

fork: create a child process

dup2: move the read fd of the pipe to 0 (stdin) in the new process

exec: run the program in the child process

There's some cleanup but that's the gist - popen isn't a syscall (or reasonably close to one), so must necessarily rely on other stuff to get its work done where it actually interfaces with the kernel.

That makes perfect sense. It's the fork+exec that makes it dangerous (e.g. in the CGI context). This is something I've been wondering about and you have just provided a jolt of clarity. Many thanks, again.

Happy to help :)

I think I previously had all the pieces required to do this, but only just figured out that it worked:

  Glenn-Willens-MacBook-Pro:~ gwillen$ exec 3<> /dev/tcp/<website>/80
  Glenn-Willens-MacBook-Pro:~ gwillen$ echo "GET / HTTP/1.0" >&3
  Glenn-Willens-MacBook-Pro:~ gwillen$ echo "" >&3
  Glenn-Willens-MacBook-Pro:~ gwillen$ cat <& 3
  HTTP/1.1 200 OK
  Date: Mon, 10 Sep 2012 21:43:33 GMT
  Server: Apache
  Last-Modified: Sun, 06 Sep 2009 18:44:42 GMT
  ETag: "30525-0-472ed20537a80"
  Accept-Ranges: bytes
  Content-Length: 0
  Connection: close
  Content-Type: text/html

  Glenn-Willens-MacBook-Pro:~ gwillen$

One thing missing is <<- heredocs, which strip (some) leading whitespace. It lets you keep up your indentation in a shell script without passing a bunch of whitespace to the commands.

I've found

    cmd <(another_cmd)

to be particularly useful in the past for pushing generated data into something that doesn't accept - as an alias for STDIN. I'm not sure if it's a different way of expressing one of the listed items though; iirc it creates a temporary fifo

That FIFO temporary file is the file descriptor of the subprocess' stdout. You're accessing it through a pipe.

It's most useful when you want to diff the result of multiple commands, e.g.: diffing the contents of two directories:

  $ diff <(find /first/path | sort) <(find /second/path | sort)

Aha!

In zsh "=(cmd)" does that, but "=" doesn't work in bash. "<(cmd)" works everywhere... why did zsh trick me with "="?

zsh had the idea first IIRC.

I've always done

   (cmd 2>&1) > file

to redirect stderr and stdout to a file. I always thought the parenthesis were necessary, but according to this I can just do

   cmd > file 2>&1

It seems very unintuitive, though. Does anyone know the rationale behind ordering of redirects?

(That said, I'm going to use 'cmd &> file' that I just learned from now on)

cmd is going to output to STDOUT and STDERR, which are file descriptors 1 and 2. In a basic shell command execution, those route out to the console.

Reading from left to right:

cmd

Run command cmd. File descriptors: {1: /dev/console, 2:/dev/console}

  > file

Open a file descriptor for "file" and copy it into our first file descriptor (stdout). File descriptors: {1: fopen('file'), 2: /dev/console }

  2>&1

Copy the file descriptor stored in 1 (stdout) into 2 (stderr). File descriptors: {1: fopen('file'), 2: fopen('file')}

> Does anyone know the rationale behind ordering of redirects?

I can't give you an exact rationale, but my old shell scripting books from the 90's use the cmd > file 2>&1 notation. My personal belief is that it was for situations when you would do something like this:

cmd > output.txt 2> errors.txt

In the first case, you're actually executing two shells.

    (cmd 2>&1)

runs in its own subshell, and copies stderr to stdout.

    >file

is interpreted in the parent shell, and copies stdout to 'file'.

In the second instance, you run a single shell, first copying stdout to 'file', then copying stderr to the (redirected) stdout, that is, 'file'.

It's read as "cmd, stdout to file, stderr to the same place as stdout".

The network pseudo filehandles (/dev/tcp/<host>/<port> and /dev/udp/<host>/<port>) are explicitly disabled in certain distros (notably Debian). Ubuntu appears to support it.

Though not limited to just shell redirections, one of the true masters of bash (and zsh) is Larry Peek whose written/given multiple books, articles, and presentations on the topic.

Do you know why debian did that? Any links to debian-devel threads?

http://lists.debian.org/debian-user/2006/07/msg00204.html

The feature is confusing (it uses virtual files visibly only to bash, and not extant on the filesystem) and overlaps existing functionality (netcat).

Also:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464

http://lists.debian.org/debian-user/2003/04/msg01591.html

http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu

http://lists.debian.org/debian-user/2006/07/msg00204.html

I can never remember the (ba)sh redirection syntax, and I used it for decades.

Rc on the other hand is much more clean: http://rc.cat-v.org

the "cmd <<EOL ..\n..\n EOL" part has been mystical to me for ages, especially as I didn't know what to google to find out. I recently learned it was called a "Here Document". http://en.wikipedia.org/wiki/Here_document

For extra credit, try using a "Here Document" as input for 'echo' and 'cat'. Explain the results.

What is so mystifying about this?

Doing this:

"cat << EOL

line one

line two

EOL"

Would be the same as this:

"cat < file"

That is if 'file' contained this:

"line one

line two"

This doesn't work with the echo command because it does not use stdin like cat does.

`man bash` , search for "<<" see the "Here Strings" section.

You can also search https://www.google.com/search?q=bash and follow links to "Documentation"

    xargs echo <<EOL  
    > line 1
    > line 2  
    > EOL
    line 1 line 2

    $ cat <<EOL
    > line 1
    > line 2 
    > EOL
    line 1
    line 2

line-terminators get eaten by xargs

The PNG version is rendered with subpixel antialiasing, which is kinda annoying to see. Besides that, awesome job.

Is there a particular reason you'd go for the PNG version over the PDF version? It's a more suitable format for these "cheat sheet" type pages, since it'll use the platform text rendering, and also keep the vector information (meaning it'll print better if you want to hang it up next to your desk).

Where is EOF? I use that a lot in my Bash scripts. I think it should be on this cheat sheet.

I like the mention of PIPESTATUS - $PIPESTATUS[*] is at the start of my PS1

Thanks! Much more useful than the other recent alphabetic cheat sheets.

Great! Thanks for sharing, very helpful.

Very useful! Thank you for this.

Reply on news.ycombinator.com