Ask HN: Best Unix Tricks? | Hacker News Reader

Ask HN: Best Unix Tricks?

83 pointsscottjackson103 comments

HN, what are your killer Unix tips? I've been using and learning about Unix both at university and at home for almost a year, and I was wondering -- what are the things that, when some wizard showed them to you, you wrote down as soon as you could so you wouldn't forget them? Is there some trick that saves you a buttload of time? Some script that makes your job way easier? Some command that made you smack your forehead in frustration when you found out it could be that simple?

Note: I'm personally looking for tips for working in bash (specifically OS X stuff if that makes any difference), but if there's some amazing thing that only works in zsh under Debian or something, feel free to suggest those too -- it's sure to be useful for someone.

I had a quick search for previous topics on this, but all I found was http://news.ycombinator.com/item?id=103725 which had a couple of badass things in the discussion.

View on HN

It's a great kill-ur unix tip but you should never fall for it:

    :(){ :|:& };

Oh, the temptation... if you're thinking of cutting & pasting that into the shell of your machine google for 'bash fork bomb' first, I've left off the final ':' to set it off.

Anyway, on a more serious note, after an installation procedure that was a lot of work, if you want to document the whole thing use:

    history | cut -d ' ' -f 5-

To give you all the commands you executed in the last little while, cut & paste that and you have a pretty good start for a how-to.

It's not a 'great killer unix tip', but it works for me.

Edit: Another one just came to mind, the ps command has a very useful option that is quite undervalued imo:

    ps ax --forest

Further to filtering the history: I use "script" to record terminal sessions. See "man 1 script".

Up-modded. But also wanted to add a spot of thanks for the bash fork bomb tidbit. That was beautiful :)

I usually use ps -auxf for the best output on linux. unfortunately the OSX version of ps doesn't support foresting, so sad.

On OS X pbcopy and pbpaste are useful for working with clipboard data. Copy data from a web-page or whatever, manipulate it on the command-line to get what you want. Putting reasonably large files in the clipboard can be useful too.

For example:

  curl http://news.bbc.co.uk/ | pbcopy # copies a web-page to clipboard

Depending on what you are doing it can be very useful:

  pbpaste | sort | pbcopy # sort the contents of the clipboard.

Another useful feature of OS X is open, which uses the finder to open a file:

  open file.pdf

  open music.mp3

The equivalent Linux commands are:

    # xclip:
    curl http://example.org | xclip

And instead of "open", you can use:

    xdg-open file.pdf
    xdg-open music.mpe

(or even gnome-open if you're on gnome) will open up the file with their default handler

And "open" works on bundles, allowing you to avoid descending into the /Contents/... part.

    open /Applications/FooBar.app

open also takes the -a argument to specify which application to use. It's very handy for things like this:

    alias shop="open -a Adobe\ Photoshop\ CS3"

You can also specify the app to open using the -a flag and specifying the full name of the app (with correct spacing and capitalization), e.g.:

    open -a 'TextEdit' foobar.txt
    open -a 'TextMate' foobar.txt

I also find it useful to have multiple instances of certain applications sometimes:

  open -na safari

open -a finder <dir> is a pretty handy way to open a finder window to current directory.

You can just pass the directory, no need to specify finder.

  open .
  open /Applications

Read man pages and "The Unix Programming Environment" by Kernighan and Pike. It's old, but conveys the design principles behind Unix, particularly how to write little tools that work well with all the others. "The Art of Unix Programming" is also good, despite Eric S. Raymond's getting on a soapbox all the time.

The 4.4 BSD supplementary books (PSD, USD, SMM, etc.) are full of major documentation, with chapters on vi, awk, cc, yacc, gprof, etc. They're very old-school. (OpenBSD installs them with its default documentation, I think FreeBSD does as well.) Some parts will be obsolete, but the big ideas are the same. For example, it covers lex and yacc better in about ten pages each than the O'Reilly book could in twenty times that. Same with make, and several other major parts in the Unix toolchain.

For starters, learn to use: apropos, cron, the basics of sed and awk, the shebang, a real editor, screen or tmux, tar, gzip, and find. Read man pages, see where they lead you. Try to automate stuff, and go spelunking in the infrastructure for whatever Unix distro you use.

I also found that I learned a LOT by getting away from stuff that assumed Linux. Soooo much Linux newbie stuff is written by people who only know Linux (and may use GNOME for most things, really). You don't really learn Unix that way. Also, watch out for people who assume that bash is the standard shell. (A pet peeve of mine.)

cd - (Go back to previous working directory)

sudo !! (Run the last command as root)

:w !sudo tee % (Save a readonly file in VIM)

> tmp.file (Empty a file)

cmd !$ (Run cmd with previous commands arguments)

They are just some of my favorites.

That should be cmd !*. !$ would only be the final argument to the previous command, which you can also insert with alt-. (and pressing alt-. again will cycle through final arguments to earlier commands).

Adding :p to the end of a history expansion (like !$:p) will cause it not to be executed, but the expanded command line will be echoed and put in your history so you can check edit it.

M-. also takes a numeric argument (M-number or ESC number) to yank the nth word instead of the last.

zsh tab-expands history expansions, so you can start editing right there

then checkout http://www.commandlinefu.com/commands/browse/sort-by-votes

I think :w !sudo dd of=% is better. Can't remember why, though.

What is the difference with :w! ?

mkdir -p (create a chain of directories in one call)

Excellent list. I use all of those except the last, which is new to me and extremely useful--so, thanks! It's now aliased.

I find myself using the first two on an almost hourly basis.

http://www.commandlinefu.com

This one is good too: http://www.shell-fu.org/lister.php?top

http://www.makeuseof.com/tag/4-websites-to-learn-cool-linux-...

http://www.hiraeth.com/alan/tutorials/courses/Unix-Tools.pdf

and Powers / Peek' oreilly book "Unix Power tools" on google books

================================

IBM had a good series:

http://www.ibm.com/developerworks/linux/library/l-11sysadtip...

https://www.ibm.com/developerworks/linux/library/l-10sysadti...

http://www.linuxformat.co.uk/wiki/index.php/58_Cool_Hacks

http://www.ibm.com/developerworks/aix/library/au-productivit...

My tip: pushd without an arg pushes the current directory onto the stack, and then switches to what popd would have. Hence, repeated invocations of pushd let you cycle between two directories without thinking.

$CDPATH is also amusing:

  $ mkdir /foo/bar/baz
  $ cd baz
  bash: cd: baz: no such file or directory
  $ export CDPATH="/foo/bar"
  $ cd baz
  $ pwd
  /foo/bar/baz

could you give an example of the pushd thing you mentioned?

    $ pwd
    /home/jrockway
    $ pushd foo
    $ pwd
    /home/jrockway/foo
    $ popd
    $ pwd
    /home/jrockway
    $ popd
    error: directory stack empty
    $ pushd
    error: directory stack empty
    $ pushd foo
    $ pwd
    /home/jrockway/foo
    $ pushd
    /home/jrockway
    $ pushd 
    /home/jrockway/foo
    $ pushd
    /home/jrockway
    ...

You can also do this with "cd -"

1. Check out z: http://github.com/rupa/z. It's a pretty nice productivity booster for flipping between different directories. It's quite easy to port to zsh, also.

2. Learn to use the editor modes on your shell. If you use Emacs mode in zsh, for example, things like transient-mark mode work with all the standard Emacs keybindings, and you can easily copy and paste pieces of your command line. vi mode also supports similar bells and whistles.

3. In zsh, if you are typing a command and then realize you need to look up its man page, move your cursor back to the command and hit Esc-h (or Meta-h or Alt-h, depending on your terminal). Once you quit the man viewer, your command line will be restored.

4. In zsh, if your command line gets too unwieldy to edit in zle, just hit Esc-e (Meta-e). It'll let you edit the command line in your editor (be sure your EDITOR environment variable is set).

5. zsh has an awesome buffer stack feature. Imagine you're typing a long command (call it command A), then realize that you need to run something else first (call it command B). Don't Ctrl-u or delete anything; leave your current command alone, and hit Esc-q. You'll get a fresh prompt. Run B. Once B executes, zsh will restore the command line from A. You can do this as many times as you want.

Anyway --- go and read "From Bash to Z Shell, Conquering the Command Line" by Kiddle, Peek, and Stephenson. I've never seen a better book on using shells with maximum efficiency (and I'm still only learning). Just learning to use and extend the autocompletion features of zsh was invaluable to me.

One of the little things that it took me a while to find out about:

  echo '2^16' | bc
  65536

'bc' is a complete calculator language interpreter that also functions quite nicely as a simple calculator. If you need to do any heavy lifting check out: http://en.wikipedia.org/wiki/Bc_programming_language

As a follow-up to that: It might seem basic but understanding pipes (the '|' symbol) is very useful! The pipe symbol 'sends' the output of one command to another. Examples:

  tail -n 500 log/production.log | less (display the last 500 lines of your production logs one page at a time)

  cat config/routes.rb | wc (count the number of words and lines in a config file)

A lot of the tricks here are using the shell builtins. Pretty much all the nifty stuff you can do with that are in your shell man page. If you are using bash, try typing man bash. Read it. There is also stuff in there about scripting bash. This is worth knowing, because short one liners are very nice for moving a bunch of files around or whatever else. I use one liners every day.

Make sure you read about job control in your shell. It helps understanding the why's and whats of some of the other tricks.

Learn to use pipes. On the cli, I use Bash, sed, awk, and grep a lot, and with pipes you can do amazing things relative to the windows world. It's a very useful skill.

Learn a scripting language. I tend to use perl because I can pipe into it or break into a move involved script, whichever I prefer. I personally find Python less suited for this, but I've heard that Ruby is a good sysadmin language too. You don't have to learn how to be a grand wizard, but any serious unix person should know one. It helps greatly with productivity.

Edit:

Forgot to add, learn find (man find). This, in combo with xargs, is very, very useful.

I wrote a bash function yesterday which allows you to

    cd ...

to move up two directories instead of having to

    cd ../..

Want to move up 4 directories?

    cd .....

Pretty useful, imo. Check it out

http://blog.jerodsanto.net/2009/09/cd-up-up-up/

In zsh, you (1) don't need to write a function, and (2) you don't need to type "cd" to switch directories.

  setopt auto_cd

  alias -g ...='../..'
  alias -g ....='../../..'
  alias -g .....='../../../..'
  alias -g ......='../../../../..'

Then you can just type "..." and go up two directory levels, and so on.

That's pretty cool. I've shied away from zsh for some time as I'm very comfortable with bash, but a lot of evidence has been coming to me lately that makes me want to give it a go.

I have just an alias for this (without the cd) in my ~/.bashrc:

  alias ..='cd ..'
  alias ...='cd ../..'
  alias ....='cd ../../..'
  alias .....='cd ../../../../'

Yah that's a nice way to fix it as well, but I prefer to be able to continue using the "cd" command since it is burned into my brain.

Can't remember where I found this, but this function allows you to go up any level of dirs without specifying an infinite number of aliases.

Usage: up {x} where x is the number of directories you want to go up (e.g., 3)

function up() { local arg=${1:-1} local dir="" while [ $arg -gt 0 ]; do dir="../$dir" arg=$(($arg - 1)); done cd $dir >&/dev/null }

readline is amazing, for bringing consistent shell-like behavior in a variety of programs, including the shell, python, mysql, etc.

You configure it using ~/.inputrc...here are the commands I use:

  set completion-ignore-case on

Tab-completion will ignore case if it's not ambiguous...

  set show-all-if-ambiguous on

...but if it is ambiguous, this will give you a list if you hit tab again.

  Space: magic-space

Magic space is key. When you press space, it will expand out any ! commands you have on the line. If you have none, it's just a normal space. This lets you be much more bold with your use of history commands, as you can see what it expands out to before you run it. Note that this can interfere a little within other environments, such as python.

  "\e[A": history-search-backward
  "\e[B": history-search-forward

These two set incremental history search. You can type part of the command you know is in your history, and then pressing up or down will cycle through them. E.g., if you're looking for that particular find command you did a few days ago, you can type 'find' and then hit up arrow until you find it.

  set editing-mode vi

Finally, not for everyone, but for us vi users, this is awesome. You get a lot of the standard vi functionality on the command line (and anywhere else you use readline, include mysql and python). This means esc + j/k to scroll up or down through history (far faster than arrow keys), using f/F/t/T commands, d/c/a/i ways to enter edit mode, even . and , work as expected, etc.

  set editing-mode vi

BTW, for those who want to use it occasionally in bash, vi editing mode can be enabled interactively by running

  set -o vi

And this command will revert it back to default ("emacs") editing mode:

  set -o emacs

Using the .ssh configuration file to save host specific options (~/.ssh/config). Never again long ssh commands to connect to your frequently connecting list.

    Host home home.example.com
      HostName home.example.com
      Port 12345
      User dryicerx
      IdentityFile ~/.ssh/somekey.pem
      LocalForward 59001 localhost:5901 
      
    Host *amazonaws*
      User root
      IdentityFile ~/.ssh/yourawskey

now on your command line, and the settings will automatically be applied.

    ssh home  
    ssh 1-2-3-4.blah.amazonaws.something.com

I specially love the last one if you are working with EC2, as it catches any EC2 domain name and applies the key with the root user.

Named bookmarks and a jump-to function:

  function bookmark () {
    echo -n "Name of bookmark (no spaces): "
    read BKNAME
    if [ -e ~/.bkmarks/$BKNAME ]
    then
      echo -n "Overwrite? "
      read CONFIRM
      if [ "x$CONFIRM" != "xyes" ] && [ "x$CONFIRM" != "xy" ]
      then
          return 0;
      fi
    fi
    echo -n "Description: "
    read DESC
    ln -s `pwd` ~/.bkmarks/$BKNAME
    echo "`pwd` : $BKNAME : $DESC : `date`" >> ~/.bookmarks
  }

  function go () {
    if [ -x ~/.bkmarks/$1 ]
      then pushd ~/.bkmarks/$1; cd `pwd -P`; 
      else echo "~/.bkmarks/$1 not found";
    fi
  }

To copy/move/rename files with suffixes quickly:

  cp /home/foo/abc.cpp{,-old}

equivalent to

  cp /home/foo/abc.cpp /home/foo/abc.cpp-old

Braces in general are great for all sorts of tasks. The general principle is that it duplicates arguments, with changes. Thus,

  foo/bar/{a,b,c}

expands out to

  foo/bar/a foo/bar/b foo/bar/c

and you can also use {1..9} to expand to all the numbers between 1 and 9 (inclusive).

Here are some ways I commonly use it:

- Create lots of subdirectories at once:

  mkdir -p test/{a,b,c}/{1..4}

- Rename/copy files (as in parent comment)

- Removing lots of randomly named files, with common elements:

  rm {a,d,f,gr}*

(this will remove all files starting with those prefixes...note that it will not delete other files starting with 'g', only those with 'gr')

ctrl + r

Starts a search through your history file.

And then ^o to execute that command, and put the next command from your history onto the command line ready for another ^o...

Thank you so much! No more `history | grep cmd` %)

Map <up> and <down> to do the same in .zsh:

  # search history on <up> key instead of paging previous commands:
  bindkey "\e[A" history-search-backward
  bindkey "\e[B" history-search-forward

Or in bash, using .inputrc (readline):

  "\e[A": history-search-backward
  "\e[B": history-search-forward

and crl+r again for the next match. The ctrl+o tip didn't work for me though (OS X).

This works because you're in emacs mode, which gives you a bunch of other tasty shortcuts (http://www.scribd.com/doc/985254/Bash-Emacs-Editing-Mode-rea...) as well. There is an equivalent vi mode as well ("set -o vi").

C-x C-e to open up the current command in $EDITOR. For previous commands, fc N M where N and M are numbers or search strings (opens the range of commands in your editor).

  set -o vi # vi like behaviour of the command line.

Use "/" to search through the history, "y" to copy, "p" to paste etc..

  cat > somefile.txt <<EOF # quickly creates a text file

Type EOF in a new line to finish.

Or just omit the <<EOF and press Control-D to finish (assuming you have not done anything strange with stty).

touch somefile.txt

His "quickly create a file" also adds text to it.

I use this to synch a directory with another:

    tar cf - * | (cd <dest-dir> ; tar xf - )

Also look at "rsync".

And I do use rsync when I want to do a differential synch.

However, the tar solution works better for large file sets, and unlike using a simple `cp -rfp`, hardlink inodes are preserved.

...also have a look at cpio's -p mode, very old-school :-).

screen is infinitely useful.

Yes, it is. I recommend remapping the standard C-a to C-z, as C-a acts as "jump to the beginning of line" in couple of shells, Emacs and Emacs clones.

I use ctrl-_ (aka ctrl-7).

I'm not sure that C-z is a great choice as it's often used for job control. I use C-o myself. I suppose you could always use C-z C-z if you wanted to suspend your current foreground process.

I settled with C-z as it is easy and fast to type with one hand. However I am occasionally too fast and hit C-z z instead of C-z C-z. 'fg' solves this.

as an emacs junkie I use backtick for my screen control. Works fine till you try to paste blocks of SQL containing loads of said backticks - easy way to fubar a screen session.

Perhaps its emacs' fault but finding an easy to use key or key combo that isn't already in use is far too hard!

tmux is good as well. Slightly different feature set, and its cleaner codebase means that it's more likely to get features that have been on screen's todo list for ages.

http://tmux.sourceforge.net/

nc lets you talk directly to a remote host, and strace allows you to eavesdrop on an arbitrary process' communication with a remote host (or with anything, really).

Actually, I rather hope there's something better than strace for the latter. But it works.

DTrace.

Also consider socat ( http://www.dest-unreach.org/socat/ ) aka "netcat++".

This is more a general hint than a single tip. Years ago I tried to improve my Unix skills by doing everything at home from the command line -- burning CDs, playing music, copying trees etc. I learnt a lot but this exercise was much less convenient and more mentally demanding than using a gui. Recently I have become much more lazy.

Funny, we're pretty much opposite.

I see my 'window manager' as a neccesary evil to run browser and email clients, everything else runs in terminal windows.

Of course you could use 'pine' and 'lynx' but that gets old pretty quickly.

s/pine/mutt/ && s/lynx/w3m/ : these get old much less quickly, though YMMV.

    xargs -d '\n'

as in

    find * -type f | egrep -i '\.(gif|jpg|png)$' | xargs -d '\n' identify

I used to use find -print0 | xargs -0 but sometimes you want to insert filters in between or the file names aren't from find.

It's not a bash tip, but if you want something that generally makes command line navigation easier without the need for heavy amounts of customization or memorization, you might consider using the Friendly interactive shell (Fish shell).

Smart history-based autocompletion, advanced tab completion, optional cd, simplified syntax, among other features.

Not as hackerish as some of the other tips mentioned here, but it's a much lower barrier to entry than the other commonly used shells out there.

Also submitted it here: http://news.ycombinator.com/item?id=811113

Add this to your .bashrc to have much more intelligent tab-completion:

  case $- in
    *i*) [[ -f /etc/bash_completion ]] && . /etc/bash_completion ;;
  esac

In addition to the usual filename completions, this will do command-dependent completions. A small sampling:

ssh: tabs complete hostnames in your ~/.ssh/config file

/etc/init.d/*: tabs complete possible options for any script in your init.d directory (at least on debian/ubuntu)

most standard linux commands: type the command and then - and then tabs will show you the list of possible options for that command.

kill/renice/pkill: tabs show you list of process ids/names

My most useful zsh functions:

  # a visual recursive list of all files and directories
  function tree()
  {
    find . | sed -e 's/[^\/]*\//|--/g' -e 's/-- |/    |/g' | $PAGER
  }

  # recursive text search (ack is also nifty)
  function f()
  {
    find . -name '*' | xargs grep -l $1
  }

  # recursive search and replace (ignoring hidden files and dirs), example: replace foo bar
  function replace()
  {
    find . \( ! -regex '.*/\..*' \) -type f | xargs perl -pi -e "s/$1/$2/g"
  }

Perhaps Python specific but I use this a lot

  epy () { 
  	cmd="import $1 as a ; print   a.__file__.endswith('.pyc') and a.__file__[:-1] or a.__file__" 
  	file=$(/usr/bin/env python -c $cmd) 
  	echo $file
  	emacsclient --no-wait $file
  }

  !!:s/sl/ls

executes last command but replaces first occurence of sl with ls. (can also be written as ^sl^ls)

  !!:gs/sl/ls

same as above, but now global replace

I have that one aliased as sl2ls ...

I alias maek to make

Why wouldn't you just alias sl to ls? It'd be much less typing.

Similarly, !! refers to the last command, so you can write "sudo !!" to execute the last command as root. Helpful for apt-get, among other things.

!$ refers to the arg to the last command, so you can write:

  $ ls -l /foo/bar
    # .oO(aha, it's a stale link i don't want)
  $ rm !$
  rm /foo/bar
  $

Pipes, grep and such come in handy in many applications.

For example:

ps -e | grep processname

will grab the pid of any process so that you can subsequently kill it.

sudo !! (run last command as sudo) is great too when you forget to type sudo on a regular basis like me.

will grab the pid of any process so that you can subsequently kill it

I like to use:

killall processname

And that works just dandy, until that one day you're on a Solaris machine...

pkill and pgrep are useful too!

altgr-sysreq-r, then -e -i -s -u -b (spells busier backwards). This will do a safe shut down and reboot including sync, even sometimes when ctrl-alt-del won't work. Also you can use altgr-sysrq-k like ctrl-alt-bksp to kill the window manager. Try altgr-sysrq-h from a console to see a terse help message.

Note altgr is the right hand alt key, the left one should work too but I find the combos harder to do then, and sysrq is "system request" the same key as "print screen".

IIRC Ubuntu now defaults to disabling the "magic sysrq" key combinations so YMMV.

grep -l textregex fileregex

print the names of all the files satisfying fileregex with text satisfying textregex. Nice if you need to find out where a function is defined.

GNU Global integrates well with the shell and allows you to do such neat stuff as "less -t func" to see the definition of func.

Process Substitution in bash

comm <(sort file1) <(sort file2) (comm file1 and file2 without temp files)

I love the '-exec' flag for find

Me too. Find also has -delete, so you don't have to type -exec rm -rf '{}' ';'

It's worth noting that -delete is more efficient than the above, and indeed safer.

Efficient because additional exec's are removed, for the most part find can perform the unlink itself. Safer because a replacement version for rm cannot be inserted into the user's path.

Before -delete was added to find I always found perl faster:

  find2perl scratch -type f -size 0 -eval 'unlink' | perl

  find2perl scratch -type f -size 0 -exec rm {} \; | perl

The last case does not actually call rm but is hard-wired to use unlink().

It may still be faster.

...except, that you should rarely use it. :)

It is much, much faster to pipe commands to "xargs". For instance:

    find . -name core -print0 | xargs -0 rm -f

Since "find" doesn't know when your command is multi-file, it has to run the command once per file; whereas, "xargs" can run it as few times as necessary. It works well for any command that accepts "a list of files".

In the example above, the -print0/-0 are a safety net to null-delimit filenames so that it doesn't matter if any of the paths contains a space.

Also the -n and -P options to xargs. So you would do something like

    $ find . -name 'blah*' |wc -l
    2000
    $ find . -name 'blah*' |xargs -n 250 -P 8 ./mangle

That'll spawn 8 copies of mangle, each with 250 blah files on its command line.

...except, when I want to execute a command once per file. :-)

If I want this I have the choice between: '-exec sed -i s/a/b/g {} \;' in find or '| xargs -IX sed -i s/a/b/g X'

Not really a trick, but something not everyone knows

appending & to run as a background process

./some_app &

man man

    import kremvax /net /net

Plan9 doesn't need nat

keep hacking on what not work, and diving into detail. I began my Linux life exactly one year ago, when I bought myself the 1st laptop. It's rewarding and joyfull

    > make me a sandwich
    what? make it yourself
    > sudo !!
    okay

Reply on news.ycombinator.com