Unix Commands I Wish I’d Discovered Years Earlier

You should do a screencast of such infinitely useful classics as tac, tsort, and sl.

(Actually, I have found use for those first two on occasion. ;)

Great links, though I admit when I saw the OP I was expecting content like that and was pleasantly surprised that they really were "Unix commands I wish I'd discovered years earlier", and not the standard array of unix tips+tricks that you can get by without for awhile but as you become more advanced become second-nature.

"Man ascii." The number of times that I wound up generating my own ascii table from whatever language I was working with...

"xxd" I always either used search in emacs hex-mode (which I find cumbersome and annoying but haven't taken the time to customize), or rolled my own search using a programming language. This would have come in handy SO many times back when I used to muck around with videogame save files and the like. I never even knew it existed nor how to discover that it did.

'cal' I learned about early on so never missed it and the 'ssh' stuff kind of fits into the regular "unix tips and tricks" category.

i usually click those links, see the stuff and roll up my eyes. nothing learned, but upvote so more people learn about them anyway.

this time, the very first one `man ascii` got me. Genius. Never though about looking for that in man before. But in my defense, this was added in 1999 :)

FWIW, the double-star recursive globbing was added in Bash 4.0. My reading of the manual[1] suggests it is disabled by default, and can be enabled by setting the 'globstar' shell option. I believe it's enabled by default in zsh.

[1]: https://www.gnu.org/software/bash/manual/bash.html#The-Shopt...

Re: command line globbing with globstar -- is there some way to get it to cache filepaths that are frequently accessed? It's quite slow for me (I usually use aliases for that, which I was contemplating getting rid of).

Thanks a lot for wising me up to the new hotness offered by bash 4+ (zsh is also pretty slow for me for some reason, even though people keep telling me to switch to that. Maybe I'm more impatient than most folks).

May I ask, what do you gain by setting it to +1 cores?

If you like xargs, but want more flexibility, I'd highly suggest GNU parallel. Such flexibility includes running jobs on multiple computers, running intensive command using all available CPU's (like xargs -P), and creating unique scripts to handle multiple parameters.

http://www.gnu.org/software/parallel/man.html

Gcal will highlight the current date for you.

http://www.gnu.org/software/gcal/

Hmm, the day is already highlighted for me (inverted color).

Still about ssh :

"typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome. "

Sounds awesome, but I really didn't understand what he means. Is he talking about using a certificate (-i option) to log in ?

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.:

  % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\)

Or:

  % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems.

  for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza; 
    do 
    ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts"; 
  done

This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a for loop and scp to push the files out and then run md5 to verify they are correct. It was a hack but useful for a small set of machines.

ps. if you are interested in learning about puppet, I have put together a screencast about it @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag...

I think most people know about this, but it's also good to know about scp. You can easily copy files using a ssh connection.

'screen' is absolutely crucial with ssh, too.

sh.py leverages this in command baking http://amoffat.github.io/sh/#baking

On X11 systems, there's also xclip:

    xclip -o -selection clipboard

I first ran into clipboard copy/paste on Mac. Discovered "xclip" on Linux. Aliased copy to 'xc' and paste to 'xp', later added 'xpc' to paste from the secondary (application) clipboard.

Makes all sorts of hackery possible.

In 2005 I wrote similar utilities for windows called ccopy, ccut & ppaste. I recently dug them up and put them on GitHub. https://github.com/FigBug/ccopyppaste

Which Unix includes that command?

Wow, I use it (almost) daily. Especially useful with the '-u' flag or uniq.

One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this:

  # Deflection    Col-Force       Beam-Force 
  0.000              0              0    
  0.001            104             51
  0.002            202            101
  0.003            298            148
  0.0031           104            149
  0.004            289            201
  0.0041           291            209
  0.005            104            250
  0.010            311            260
  0.020            104            240
  

You can chain commands together to quickly get ball park figures. I do this all the time when reviewing log data to get simple counts and look for anomalies.

  $ cat data.txt | grep -v '#' | awk '{print $2}' | sort | uniq -c
      1 0
      4 104
      1 202
      1 289
      1 291
      1 298
      1 311

Let me break this down a little. I cat data.txt, which just prints the contents to the console, I use grep to remove the # header, then use awk to print the second data column, the sort the values for my next procedure, and lastly I use uniq -c, which takes the input and counts all the unique values. So you can see that "4 104" means that 4 instances of 104 were found. I use this often to look at httpd logs for IP addresses or to review sudo commands, etc. Very handy!

ps. there are optimizations that could be done here, but that what is great about unix and "one function" utilities. There are many ways to the same destination!

I find it fascinating that someone could be using something as complex and slow as Hadoop and not know of sort(1). Not faulting any lack of awareness, but it is really interesting that Google map/reduce and Hadoop marketing (word of mouth?) are so effective while the contents of /usr/bin are like buried treasure.

The funny part is that one of the early uses of Google's map/reduce approach was probably to distribute the job of sorting search results among many servers. And here you are sorting Hadoop results with something as small and simple as sort(1).

pushd and popd are very useful too.

Wow. Thanks for this. I always cursed myself for forgetting to pushd, and now it turns out I didn't need to!

And `git checkout -` to checkout the previously checked-out branch.

Also in that vein on Mac it's worth having a look at Alfred http://alfredapp.com

I second the Quicksilver recommendation. When you combine Quicksilver triggers (keyboard shortcut available regardless of what application you're in) with applescript, and fabric it's very powerful. I have a trigger that that restarts all of my local services using fabric and opens up new terminal windows running tails of the relevant logs.

The Applescript is this if anyone wants to do something similar:

tell application "Terminal" do script "tail -F /somepath/some.log" do script "tail -F /anotherpath/another.log" do script "sudo fab -f '/pathto/fabfile.py' local deploy" activate end tell

Here's a detailed explanation of setting up Triggers:

http://leafraker.com/2007/09/17/how-to-create-a-quicksilver-...

Yup, cool use of named pipes.

I kind of like how the hex and char columns are next to each other though. Makes it easier to read those two columns together quickly.

baobab ?

And find, that's a nice alternative to xargs when your args are a huge set of files.

xxd is on OSX, but not on FreeBSD. Under FreeBSD, I always use 'hd somefile.bin'.

That's how I inspect a file in Vim: :%!xxd

It's even listed as a 'Convert to HEX' item in the Vim GUI menu.

I never heard of it, untill today.

I can't remember how many times I implemented hexadecimal dumps and undumps because I didn't want to coerce od into working the way I wanted.

`mdfind' in non-Mac UNIXs is spelled `locate' (and the corresponding `updatedb' updates the indexing db, obviously (although you don't have to do this often, as it runs as a cron job)).

and for readers age 30-40+:

Summary: man ascii, cal, xxd, ssh, mdfind

:-)

Counting number of files is probably not a sufficiently frequently used case to justify a new standard command. What part of piping concerns you?

> Can't they just directly access inodes for high speed counting?

Don't know what you mean by that exactly. But, I'd be stunned if find is actually reading the actual files rather than just reading the directory entries in directory "files". Which is what I'd call accessing inodes "directly"

tree does this, don't know about suppressing the actual tree, but it tells you number of files and dirs printed at the end.

It's not usually installed by default, but tree will do this: http://linux.die.net/man/1/tree

Available in most repositories, source code available at: http://mama.indstate.edu/users/ice/tree/

There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next pass. If you do this before that data gets removed from the cache it will be much faster.

  imac:~
  $ time find . | wc -l
  419191

  real	0m37.865s
  user	0m0.385s
  sys	0m2.611s
  imac:~
  $ time find . | wc -l
  419195

  real	0m10.503s
  user	0m0.330s
  sys	0m1.430s

If you're dealing with a truly large number of files you could use "find . -type f -printf 1 | wc -c" which would just print a single character to the pipe and count the characters. I really doubt this is significantly slower than one program that does both would be. It'll do roughly the same amount of work.

Try something like:

find . | sed -e 's=/[^/]*$==' | uniq -c

It's not the pipe through wc, but statting the directory entries which slows you down.

If your locatedb is current, you can use locate for this:

    $ locate $PWD | wc -l

That runs in just a hair under a second (0.908s real) for the first instance. 'find' takes a few seconds on first pass (it's got to read from disk), but is surprising fast (0.227s real) on subsequent runs.

So I guess it depends on whether you're doing it just once or multiple times.

I have this in my .bashrc:

function lc() { if [[ "$#" -gt 1 ]]; then for DIR in "$@"; do echo -n "$DIR - " ; ls -AU1 $DIR | wc -l ; done ; else ls -AU1 "$@" | wc -l ; fi; }

Then "lc dir/" will print the number of files in the directory, and "lc dir/*" will print the number of files in each of the subdirectories. This exactly what you're looking for because it doesn't descend into the tree. If you're working with directories that may have tens or hundreds of thousands of files in them, it's useful to check first.

du just uses the fts API, so if you want something directly equivalent (omitting options and error handling) it was only barely too long to post: http://pastie.org/8315296

Compared to the equivalent find, code #1 is much faster in this completely scientific best of several runs:

  $ time (find OSS/llvm -type f -or -type l | wc -l)
  real	0m4.681s
  $ time du -d0 OSS/llvm
  real	0m4.863s
  $ time ./fts OSS/llvm
  real	0m0.559s

Ostensibly because of the lack of stat calls. On the other hand the second paste and find with no arguments are equivalent.

edit: screwed up benchmarks

Aliasing to 'cal -3' can also be good if you want a little continuity.

you can even look up `cal sept 1752`

Right, you can use this to easily find the number of days in an arbitrary month in a script:

  % cal 2 1972 | tail +3 | wc -w
      29

Ok, that is pretty freaking awesome. Thanks.

For others who don't know it: http://en.wikipedia.org/wiki/Pushd_and_popd. It essentially is a stack for the working directory, so you can get back to where you were previously quickly and easily.

For OpenBSD ksh I've got this hack

  alias d='[ ${#DIRSTACK[@]} -gt 0 ] && for i in `jot ${#DIRSTACK[@]} 1 ${#DIRSTACK[@]} 1`; do echo ${DIRSTACK[$i-1]}; done; echo $PWD'
  alias popd='[ ${#DIRSTACK[@]} -gt 0 ] && { cd "${DIRSTACK[${#DIRSTACK[@]}-1]}"; unset DIRSTACK[${#DIRSTACK[@]}-1]; }; d'
  alias pushd='DIRSTACK[${#DIRSTACK[@]}]="$PWD"; cd'

zsh has AUTO_PUSHD, guess what it does.

I have my Bash set to vi keybindings, but I had no idea it was possible to remap them. Now I can finally stop hitting escape in Bash too. Thank you so much!

I use tr all the time when I want to check $PATH:

echo $PATH | tr : '\n'

If you don't want to clog up your terminal, try 'less' and then hit 'F' to follow.

The origins of xxd seem to be a bit murky, as I've seen others say the same thing about xxd being available only if vim is installed, but I've never seen a Unix system that didn't have vi(m) included.

    $ man xxd | grep vi
       Use xxd as a filter within an editor such as vim(1) to hexdump a	region
       Use xxd as a filter within an editor such as vim(1) to recover a	binary
       Use xxd as a filter within an editor such as vim(1) to recover one line

http://grail.cba.csuohio.edu/~somos/xxd.c

From what I've been able to find, vim was released in 1991, and xxd has been around since at least 1990, whereas vi was written by Bill Joy in 1976. (according to Wikipedia)

Interesting bit of trivia, perhaps someone with a good memory can clear it up for us?

For those of us using emacs-style bash keybinds. For vi-style keybinds, it's something more like "esc dd".

PS if you are a Vim user, you owe it to yourself to try the vim-style keybinds. They make a world of difference.

Actually, after hitting C-R <keyword>, you can keep pressing C-R to search backwards through the history. Or C-S to search forwards. More info: (bash uses emacs editing mode by default) http://www.gnu.org/software/emacs/manual/html_node/emacs/Bas...

>I also use ctrl R <keyword> at the bash prompt but that only gives me the most recent match. //

Are you sure, I'm using BASH on Kubuntu and I get previous matches by repeated use of ctrl+R.

Actually inspired by a previous post here I made a script "hs" that searches archived history files and shows me all past match lines. Before that I did what you do now.

Try C-x C-e – especially helpful if you want to paste stuff from websites.

nice! did not know that.

the last time one of these came up I learned about bash's vi mode:

    set -o vi

it's awesome

There are a lot of usecases for bot of them, mostly debugging I's say. So I'm gonna tell you what I used them for in the past months:

The ASCII table might be useful when you're dealing with encoding problems and want to know e.g. what exactly happened in a broken URL encoding. I used both of them for binary exploitation in the security class I took at university: When utilizing buffer overflows, it's important to get every byte at the right place of your payload. Sometimes, you also need to treat it as a string and that's where ASCII comes in.

These are only two specific usecases. As I said, there are a lot more. Development of binary file parsers could be another.

This is a very silly use, but I use `ascii` when I can't distinguish between 0 and O in a font ? `ascii 0`, for example, clears that right up.

Yup, it's a great way to review a lot of commands without reading every man page.

Want to find all the tools on your system that use the curses library (for user-friendly interfaces)? Try this:

  for i in `\ls /bin/* /sbin/* /usr/bin/* /usr/sbin/* | sort -u` ; do ldd $i 2>/dev/null | grep -q curses && whatis `basename $i` ; done | grep -v 'nothing appropriate'

Don't know how your current path finds a program? Use which:

  pwillis@zippy:~$ which df
  /usr/bin/df
  pwillis@zippy:~$ which mount
  /bin/mount

Annoyed that /sbin and /usr/sbin aren't in your path? Use whereis:

  pwillis@zippy:~$ which ifconfig
  which: no ifconfig in (/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib64/kde4/libexec:/usr/kerberos/bin:/usr/lib64/java/bin:/usr/lib64/java/jre/bin:/usr/lib64/qt/bin:/usr/share/texmf/bin:.)
  pwillis@zippy:~$ whereis ifconfig
  ifconfig: /sbin/ifconfig /usr/man/man8/ifconfig.8.gz /usr/share/man/man8/ifconfig.8.gz /usr/X11/man/man8/ifconfig.8.gz

Brace expansion is great! My understanding, though, is that you'd need to write:

  cp file{,.bak}

Was it a typo, or is there some feature/special-case I don't know about?

you can use telnet as an alternative even though it is not secure like ssh.

There are a large percentage of people who first encounter unix on single-interactive-user machines.

Wow, now I feel old. I thought everyone got started with SunOS on a Sparcstation 2? Seriously, don't most people encounter it either through installing a Linux distribution or firing up a terminal on OSX? Or is even my-first-Unix a virtual experience?

There are free unix shells out there, google will find them. Don't expect anything fancy, but you will be able to try all the commands talked about here.

To connect to the shell from Windows, you can use the Putty program (google for it), although the shell documentation will almost certainly tell you how to do it.

Thank you.

Yes it is!

Indeed! Apropos is our friend.

My favorite part is when you implement something you could've had running in a minute or so, but instead developed it for hours or days and it's still not as good as the existing solution. I often get that with C and libraries. There has to be a better way to search through existing stuff which is good and proven, not just for C of course.