Advanced Bash Scripting | Hacker News Reader

Advanced Bash Scripting

tldp.org

120 pointsmbowcock34 comments

This is hands down the best Bash resource online or in print. Don't let the "advanced" throw you off. It's the source of pretty much everything (of significance) that I know about bash.

For other applications it makes sense to use your scripting language of choice (libraries, etc.) but in any situation where you need absolute portability or availability on an otherwise bare system, bash is the way to go.

There are whole projects like the Arch Linux Installation Framework (AIF) written in bash for this reason.

Named arrays, quick string handling and the [[ ]] test syntax alone make this worth reading, and that's just scratching the surface.

> in any situation where you need absolute portability or availability on an otherwise bare system, bash is the way to go.

Interestingly, this is why Debian uses dash: A smaller shell is more portable.

How is that portable, across different architectures?

I have to work in Solaris, SUSE SLES, some other linux where I don't know what they run, and OS X and I don't have root on most of those. bash seems to work the same everywhere and always seems to be there when I get access to a newly provisioned environment.

Not every Unix-like OS comes with bash, and certainly not by default. In particular, Solaris ships with a particularly crippled /bin/sh, although they do have a proper POSIX-compatible Bourne shell at /usr/xpg4/bin/sh.

If your Solaris machines have /bin/bash available, consider yourself very lucky.

dash is portable in the sense that Debian needs it to be: they can port it to the architectures and systems that they need it to work on.

dash is also portable in the sense that you need it to be: it implements the POSIX shell spec and not much more, so it's likely that you have access to all its features when you need to use another shell as well.

If you need portability you should write your script in POSIX shellscript, not Bash shellscript.

Also on the tldp site is a 'beginner's BASH guide' - in fact there are a number of guides:

http://tldp.org/guides.html

Unfortunately a lot of the stuff has not been updated in 5 yrs or so - but it is still fun to read through from time to time.

It is not exactly a BASH guide but another good guide is "UNIX for Poets" - I think there have been a number of news.ycombinator posts regarding it. This is a good intro for the text manipulation tools you would tend to use in a BASH script.

(see http://news.ycombinator.com/item?id=2456929 )

While bash is certainly more pleasant to write and use than the other shell languages, I made myself a rule I abide by, a long time ago. Never use shell to write anything non-trivial. Unless it's a one liner that just stitches some Unix commands together, just use a real language (Python, for example). I can't count how many times I found myself being grateful for this decision. Shell scripts just have a way of growing in size and complexity, and swallowing the shell pill is the quick road into abyss.

I wrote large bunch of software in shell, so I have habit to use strict mode ("set -ue"), so shell scripts are much safer.

I write scripts like large programs from scratch: decomposition, error handling, proper logging, unit testing, packaging, etc.

I wrote my own library with few useful modules, which is designed for strict mode (bash-modules on assembla), which I recommend to use even for trivial programs.

I have no problems with bash.

Why did you do all those things? Did you have to use bash for all that software? Were you concerned about portability? Do you just dislike python?

I have no trouble understanding how complex programs can be written in bash, but I have never understood why.

Because I can. Sometimes - yes, sometimes - no, but why not? No, I am fine with Linux. Yes, I dislike python.

I hope, it is temporary problem. ;-)

This is all good, until you want your program to be really large and/or use some serious external libraries which bash lacks. Or, if you need to attach some kind of non-CLI UI to it (GUI or web), or (heavens forbid) if you need to port it to windows. Then all your plans crash and burn, and you either suck it up and port the app to Python/Perl/Ruby/C++/whatever, or just keep building an un-maintainable mess. And don't forget someone else will probably have to maintain it at some point.

It is hard to write really large and complex programs in bash, so scripts stay small, which is good thing.

In my scripts, I declare and describe command line interface of each script, so it is not hard to rewrite script in an other language when necessary. About 5% of my shell scripts were rewritten in an other language (Perl, Java, C). I had no problem with that.

Other developers and admins are happy with my software, I even got some internal awards.

My documentation/code ratio is about 7/4 (7 lines of comments and documentation per 4 lines of code), so it is easy to catch design errors - they are hard to describe in documentation in most cases. :-)

I am avoiding debugger as much as I can (lack of builtin debugger in bash helps me a lot :-)), so my software handles errors in most cases and error messages are _very_ descriptive.

I have no problems with bash, perl, C, Java, JavaScript, etc.

I recently found this site useful. I wrote a script that glued together some commands to be run by cron. Then the requirements grew and the script's complexity grew, and grew, and before long I learned that bash scripts support functions.

At this point, I wished I had started in python.

To answer maven911, I usually do LAMP stack development but pick up all kinds of technologies along the way.

Thanks for answering, I should have given some background on why I am asking this, bash is one of many technologies that I need to learn for the job - our product unfortunately uses thousands of lines of bash scripting, and I have never gotten around finishing this guide - mostly due to laziness (after work) since I don't have time during the job to look into it as we are always in fire-fighting mode.

Here's the short version of the Advanced Bash Scripting Guide: "Use Python".

Long bash scripts are less readable and maintainable than python scripts.

It doesn't have to be Python--you can use pretty much any language you want, so just use your favorite one be it Ruby or Python or Haskell or whatever.

I personally only use Bash when the script is just a trivial translation of commands I enter.

sometimes you have no choice, such as working on telecom systems which have a stripped down Linux version running on it and there is no python/perl interpreter available.

That's certainly a possibility. For me, however, the vast majority of scripts I write will only ever run on my own computer or maybe also on some AWS instances. I suspect this varies a lot based on what sorts of things you generally work on.

Sometimes you could write in Ruby/Python/Perl but it's a good idea to make a shell script instead, so that sysadmins can understand it.

Speaking as a person that converted a big shell script to Python a while back, I think both languages have their strengths. Writing something like:

    prog1 2> prog1.log | prog2 2> prog1.log | prog3

in Python is not nearly so succinct as in the shell (using only Python's standard library). And don't screw up waiting for processes or flushing buffers, things that the shell takes care of for you AFAIK.

On the other hand, things like string or date manipulation are easier in Python than in the shell.

This guide is a great read even if all you do is short bash scripts.

Actually perl makes it way easier than python, because it supports more of bash features directly, from backticks to here-docs.

I needed this guide to help me write my bash script that I use to build python and all its dependencies on all the machines I need to work on. That way I'm not at the mercy of whatever out of date python happens to be installed with the package manager by the admin.

Shell is Lingua Franca for system administrators.

I'd really like this as an Epub, Pdf or something I can just stick on my iPad. Anyone know of a link for this... or am I going to have to do this for everyone? :]

Please share if you do! epub would be perfect

If you go "one directory up", they already have PDF versions: http://www.tldp.org/LDP/abs/

has anyone gone through the entire book/guide, and i wonder is it because of your job and what other technologies you have to know in-depth

I use it more as a reference / scan it for inspiration when I'm stuck. Way better than the bash man page...

I use it like I use 'perldoc perl' or http://docs.python.org/ or https://docs.djangoproject.com/en/1.3/ or https://developer.mozilla.org/en/JavaScript/Guide or http://docs.jquery.com/Main_Page or http://www.dpawson.co.uk/xsl/sect2/sect21.html etc.

I've read the whole thing at one point in time. I was an embedded programmer for years, and the availability of bash, and the lack of python/ruby/whatever made it pretty much a necessity.

Once you grok it, it is a pretty excellent tool for testing, monitoring, coordinated pieces, etc. Besides Solaris, I can typically sit down at any type of machine or embedded device and write a fairly non-trivial script without having to ask, "what version of python is installed?" or "is it hooked up to the outside world, so I can install these libraries?".

There's power in availability and commonality even if you give up some readability.

ABS can be handy for some quick look up, but if you want to learn Bash then go with Greg's Bash Guide:

http://mywiki.wooledge.org/BashGuide

Check also the list of Bash online tutorials:

http://wiki.bash-hackers.org/scripting/tutoriallist

Thanks for the link to Gregg's BASH guide. Useful information.

hasn't this resource been around the block for 10+ years? I remember googling' "advanced bash scripting" function in college and finding a resource just like this one...

Reply on news.ycombinator.com