An Awk Primer

The O'Reilly Sed & Awk book is a lot of fun, too.

http://shop.oreilly.com/product/9781565922259.do

Something to add to your 'bag-of-tricks' -- this was once handed down to me by a master by the name of .ike, as it was handed down to him before. Maybe you'd be interested:

"The 3-finger Claw Technique" The sweetest 3 functions, ever.

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "Cannot $*"; }

and always, exit 0

I use these daily, they're extremely cross-platform friendly in any bourne-derived shell I've used in the last 4 years.

  safexample.sh :
  --
  #!/bin/sh

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "cannot $*"; }

  # consider the following lines
  safe cd /some/dir
  safe tar xzvfp /my/big/tarball.tbz

  exit 0
  --

In the above example, using 'safe' suddenly gives the user the following special powers:

1. if `/some/dir` does not exist, the script will safely exit before that un-tarring does any damage.

2. the actual 'directory does not exist' will return to stderr for the script, (as opposed to just telling us the useless fact that the enclosing script failed). Now bourne shell starts behaving like a modern language! (ala Python/Ruby tracebacks, Perl errors, etc…)

Additionally, if you 'exit 0' at the end, you can run non-safe operations, and always be guaranteed that if the shell exits, it was 'complete'. An example:

  # consider the following lines
  safe cd /some/dir
  tar xzvfp /my/big/tarball.tbz

Now, safe was removed from the un-tar command, right? So, imaging tarballs created by people using a Mac (with HFS+ and some filesystem-specific sticky bit somewhere in the filesystem which was tar'd up). Now, you un-tar it on some BSD or other NIX box, and tar complains as it goes- and exits non-zero. One way to handle this, is to consider the tar 'reliable', and not use safe for it.

Now, if we continue to be conscious of this simple safe/notsafe distinction with these scripts, they can be called by other safe scripts- and behave just like any respectable UNIX program!

Now, consider this crontab entry:

  1       3       *       *       *       someuser flock -k /tmp/safexample.lock /path/to/safexample.sh

If it fails, (for bad perms, no tarball to unpack, etc…) cron will actually have a reasonable message to email/log on failure!

Thanks to the noble efforts of the Clan of the White Lotus, these 3 functions originate during the late thirteenth century. The original author, William Baxter, explained that these three functions took 15 years to boil down to what they are- I believe that after mercilessly abusing them for several years myself…

Below is another example with more bits.

  #!/bin/sh

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "cannot $*"; }

  for i in "before $@ after"
  do
   echo "arg <$i>"
  done

  for i in "before $* after"
  do
   echo "arg <$i>"
  done

  safe echo "this is ok"
  safe echo "this is ok, too" || safe echo "so is this"
  safe bad_echo "this is bad"

  exit 0

No element of competition but have a look at The Command Line Murders : https://github.com/veltman/clmystery

UNIX Power Tools by O'Reilly is a text I strongly recommend for this. It's a sort of hardcopy hypertext narrative through a number of command-line and terminal-focused tools.

Parts of it are dated now (it's ... gasp, 22 years since its first edition), but since it's one of those books based on UNIX philosophy it's aged quite well.

It, Linux in a Nutshell, the Sed & AWK book, are still a pretty good introduction.

http://www.powells.com/biblio/61-9780596003302-0

If you've got bash, sed, grep, and add awk, then you are 80-90% of the way to Perl 4 proficiency -- that's basic Perl circa 1990, adequate for short scripts (and more efficient that bash/sed/awk scripts). Perl started life as a superset of awk, with sed, grep, and some weird extras (streams) thrown in. Indeed, the perl distro ships with a2p, a program for automatically converting awk scripts into perl scripts.

Perl 5 adds variable scoping, pointers, modules, OOP and a bunch of powerful expressive stuff and abstractions -- but if you learn awk on top of what you've already got it means you're about 90% of the way to introductory Perl proficiency, and you can add the advanced stuff later.

So by all means learn awk! Just remember that it's not suitable for really big jobs -- if you need to create awk scripts that go beyond a couple of dozen lines of code, that's when you'll probably want to upgrade to Perl.

I think there is no use to master bash, sed and awk. When you have a task that becomes tricky (beyond entry level) in sed, it is often easy in awk. When you have a task that becomes tricky in bash or awk, switch to a more powerful scripting language like perl/python/ruby/?. I love perl, but I feel it is getting outdated. I try to improve my programming speed in python but I am still far from my speed in perl.

Using Python is probably a better choice though. Its so easy to call other programs, emulate a bash script and much much more....all in code which is easily readable and maintainable!