Better bash in 15 minutes | Hacker News Reader

Better bash in 15 minutes

robertmuth.blogspot.com

475 pointswsxiaoys151 comments

If you're using:

    set -o errexit # or set -e if you prefer

Then you probably also want:

    set -o pipefail

Otherwise, it only checks that the last command succeeds, so something like:

    ls *.ssjkfle | wc -l

will actually continue as success despite the "ls" failing.

'set -o nounset' is a must have. I Just suffered this script from Samsung Printer setting Utility. sudo ./uninstall wiped the /opt

  DEST_PATH=/opt/$VENDOR/$DEST_DIRNAME
  #remove destination
  VERSION=`cat "$DEST_PATH/bin/.version"`
  if rm -fr "$DEST_PATH"; then
  	echo "INFO: $APP_NAME (ver.$VERSION) has been uninstalled successfully."
  ...

Wow, nounset or not, doing a `rm -rf` on a variable without any check is quite irresponsible. Especially if they expect the script to be run as sudo.

Just checking if .version exists and if it doesn't exiting with an error code would be a huge improvement, checking if $DEST_PATH is set is a must as well.

Scary to see a big company like Samsung release code this bad.

Another useful capability I use to do cleanup is `trap`.

    function cleanup {
        ...
    }
    trap cleanup EXIT

See more here: http://linux.die.net/Bash-Beginners-Guide/sect_12_02.html

to clean the command prompt line after CTRL-C:

    trap "{ echo; exit 1; }" INT

I don't know, if you're fearing a ctrl-c in the middle of one of those newfangled moving, colourized progress bars (hello, npm), "reset" might be more appropriate?

sure, if you do coloring/bold/underline/inverse in the script, reset would be appropriate. but I wouldn't do it by default, because it's quite slow ('time reset' takes 1s here)

My reset also takes suspiciously close to one second. Turns out it sleeps for one second at the end, probably to give the terminal time for something.

"tputs reset" has much the same effect (I think; there are some cases I've noticed it not fully restoring things) without the delay.

Is there a way to determine whether the exit occurred due to the bash script finishing successfully, or due to an error (via set -o errexit)?

You could trap the error.

Use:

  #!/usr/bin/env bash

Instead of:

  #!/bin/bash

This makes the script more portable as you don't rely on bash (or any executable) to be in /bin.

http://en.wikipedia.org/wiki/Shebang_(Unix)#Portability

Are there any situations where you wouldn't be able to find bash in /bin/bash? I haven't ever seen such a system, but I'd like to know if it's something I might run into...

The other thing is, if you're targeting obscure systems, wouldn't it be just as likely that there wouldn't be a /usr/bin/env, /usr/ might not be mounted, or that bash might not be installed at all?

I suppose what I'm asking is: for practical purposes, is the lowest common denominator /usr/bin/env or is it /bin/sh?

> Are there any situations where you wouldn't be able to find bash in /bin/bash?

Yes there are.

OpenBSD for instance installs third party software under /usr/local. Bash is not a part of the base system.

env(1) on the other hand is a POSIX standard utility and comes with the OS.

[in addition to what clarry said] Habit of using /usr/bin/env becomes more important in other use-cases, such as invoking Python interpreter.

Exactly. I often use "#!/usr/bin/env python2" to specify python 2.x on my systems where python 3 is the default.

The lowest common denominator is /bin/sh. But bash has a lot of niceties, and the POSIX shell can get the job done, but that lowest common denominator is pretty low. You're still going to be dealing with annoying differences on the platforms anyway.

I don't use env on my shebang lines specifically because it's then PATH ordering dependent (there used to be a thing where /usr/local/bin and GNU tools were last in root's path, but first in non-root users' path). I'm more confident (perhaps incorrectly) that /bin/bash or /usr/local/bin/bash is what I expect it is vs the first thing found in PATH that is named bash is what I think it is (however, this applies moreso to coreutils-like things, that have different SysV vs BSD semantics/options, vs a shell such as bash, which is known everywhere to be GNU bash). Some tools, like ssh, can propagate environment settings based on local, remote or config file settings, and I'd rather not be surprised.

This used to be a bigger deal on systems that put reasonable (where "reasonable" == "what you're used to") tools in /usr/ucb or /opt/gnu, rather than system paths. If you're going to create something that's intended to be "portable", you've got bigger fish to fry than if and where bash is available, and it's wise to abstract system differences to different scripts (run.linux.sh, run.freebsd.sh, run.osx.sh, run.aix.sh, etc) than try to keep everything in one massive script anyway.

Yes, using PATH for determining bash location is quite vulnerable to all kinds of security exploits.

  FreeBSD = /usr/local/bin/bash
  OpenBSD = /usr/local/bin/bash

OS X ships with an increasingly ancient version of bash owing to GPLv3. You probably want to use a later version installed with brew / port etc.

If you look after random mixtures of Linux and similar you'll be OK, but I've juggled OpenBSD , FreeBSD, Solaris, and Linux all at once.

Usually Solaris would get GNU tools installed beneath /opt, or similar, rather than in /bin.

I always thought the purpose of the /bin directory was for the basic unix commands.

https://en.wikipedia.org/wiki/Unix_directory_structure#Conve...

I always thought of it as the place for essential binaries needed for system rescue (single user boot). For this reason, they should also be statically linked. I got this a long time ago from reading the Filesystem Hierarchy Standard [1]

Not that this rule of thumb helps define the location of bash. It may or may not be considered essential for recovery.

[1] https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard http://www.linuxfoundation.org/collaborate/workgroups/lsb/fh...

Not necessarily statically linked, though IIRC OpenBSD's /bin/sh is for both robustness and security reasons. Debian distros offer an alternative sashroot login which uses the stand-alone shell, which is both statically linked and includes numerous additional built-ins useful for recovery or forensics work.

I have seen it in corporate environments where specific versions of things like bash (but more commonly, python, ruby, etc) are pushed to machines to places like /opt/ so that the deployed application could use them without fucking around with the package management of the underlying host. The deployed applications would then be kicked off in an environment with a modified PATH to reflect their dependencies being in those locations.

Those sort of deployments got a lot of milage out of /usr/bin/env

Something along these lines would allow using bash without making your script completely broken:

    #!/bin/sh
    if [ -z "$BASH_VERSION" ]; then
        if which bash >/dev/null 2>/dev/null; then
            exec bash "$0"
        else
            echo "ERROR: bash was not in your PATH" >&2
            exit 1
        fi
    fi

    # now start using bash-only shortcuts safely
    if [[ "hello" == "hello" ]]; then
        echo Bash has been achieved.
    fi

I'm not promising that's completely foolproof (it clearly isn't), but it works with the default Ubuntu 13.10 config of having /bin/sh linked to /bin/dash. And you could easily expand on the idea to look in specific locations for the bash binary even if it's not in the path, check those binary's versions, etc.

and it doesn't work with bash-specific shell syntax, which will error upon script parsing in POSIX shells.

Any {Free,Open,Net}BSD system.

>Are there any situations where you wouldn't be able to find bash in /bin/bash?

Basically everything that isn't a typical gnu/linux distro or osx. Bash is not a standard tool, it is a gnu specific shell. Posix requires env to exist, and requires it to exist in /usr/bin/. If you are using bashisms, then using /usr/bin/env bash will let people without bash see that your script is a bash script, and install bash to solve the problem. If you hardcode /bin/bash they have to fix your script.

> is the lowest common denominator /usr/bin/env or is it /bin/sh

Ideally you should use /bin/sh of course. But it is missing functionality some people want to use, so in that case using bash the portable way is preferred over using bash the unportable way.

The only real situation I'd like to use that is on Android, but it does not have /usr/bin/env either.

If I want to be that portable, I just use #!/bin/sh and write a straight POSIX shell-compatible script instead.

I'm glad this included the "Signs you should not be using a bash script" section. Bash is a very good solution for many cases, but it becomes downright unruly when dealing with a lot of string manipulation and more complex objects.

    your script is longer than a few hundred lines of code
    you need data structures beyond simple arrays
    you have a hard time working around quoting issues
    you do a lot of string manipulation
    you do not have much need for invoking other programs or pipe-lining them
    you worry about performance

It's not a sign you shouldn't be using bash. It's a sign you shouldn't be using ONLY bash.

People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man).

You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes.

You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution.

Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.

I think I might have accidentally down voted you. That was helpful feedback for me.

No problem... the way I think about is that Python/Perl/Ruby are for data structures, logic and control flow; while shell is for configuration and data flow.

As mentioned above, things that belong in shell, and NOT in Python:

  - file system paths (these are parameters to Python scripts, not hard coded)
  - port numbers
  - user names
  - other config values
  - concurrency -- you can get really far with xargs -P, and sometimes &
  - pipelines (try doing this in Python without shelling out; it's horrible)

Things that belong in Python and NOT shell:

  - complex control flow (e.g. I rarely write loops in bash; never a nested loop)
  - data structures (bash has hash tables, but I never use them)
  - arithmetic
  - parsing and cleaning individual lines or fields of data

Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.

File system paths may belong in the shell, but what about paths with spaces?

To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(

Here are the quoting rules I use:

  - Avoid paths with spaces :)
  - If you have to handle paths with spaces, use double quotes everywhere,
    e.g. "$pidfile".   
  - Always use "$@".  I've never found a reason to use $@ or $* unquoted.

I also write unit tests (in shell) if there is something unusual I know I have to handle.

That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple.

There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code.

I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.

> If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile".

I'd like to add: Always use double quotes around paths. Or is there something I'm missing here?

  $ ls -l "*"
  ls: cannot access *: No such file or directory

Of course, you might want that exact behaviour, but sometimes you want e.g. all .table files in a path with a space. I believe

  $ ls -l "path with/lots of spaces/"*

is the correct solution, but it seems silly. I'm not entirely sure why variable expansion works inside double quotes but wildcard expansion doesn’t.

> I'm not entirely sure why variable expansion works inside double quotes but wildcard expansion doesn’t.

That's because a double-quoted expression is supposed to evaluate to a single word. Wildcard expansion can result in multiple words, but parameter expansion doesn't.

(Except for things like "$@". With bash, the rules always have exceptions.)

TIL. Thank you very much! :)

A basic technique in bash is "UMQ": Use More Quotes

Modularity is nice, but it's generally easier to debug a program in a single context.

I work with some deployment systems that chain together small scripts in a bunch of different languages. They are a nightmare to troubleshoot.

I'd much rather the spaghetti was all on one plate than follow it from table to table...

If the tools are coherently designed, it should be easier to debug, because you can just use -x to log the commands being run and paste them in to see what went wrong. It's debugging with a REPL.

The biggest mistake I see people making is to hard code paths, ports, user names, configuration, etc. inside Python/Perl scripts. All that stuff belongs in shell. All the logic and control flow goes in the scripts. If it's factored this way, then you have a very natural way of testing the scripts with test parameters (i.e. not production parameters).

I don't doubt that there are many multi language shell scripts that are spaghetti. Factoring into processes is definitely a skill that takes thought and effort, and I don't think anyone really teaches it or writes about it. The only books I can think of know of is The Art of Unix Programming by ESR and the Unix Programming Environment.

But it's definitely made me way more productive once I started thinking like this. People say talk about the "Unix philosophy" for a reason. It's a real thing :) It's not an accident that Unix is wildly popular and has unprecedented longevity.

I'm glad someone mentioned debugging. Typically Bash scripts evolve into programs and one of the first things I always notice is how much effort I'm having to put into debugging. Indeed, since I started working with [plumbum](http://plumbum.readthedocs.org/en/latest/) I now typically reach for python in favour of bash even for small jobs.

Yup, until someone can show me a bash editor that understands all the processes that are invoked and gives me at least context aware code completion, parameter documentation, "go to definition" and "find all references" I'd rather keep my code in a language where I can get those features.

This is how I work. Whenever possible, when I have to write a utility in whatever language, I write it such that it can easily be integrated into shell pipelines. That makes it far easier to glue multiple tools written in different languages together.

> …awkwardly and verbosely reimplements 'xargs -P'…

Verbose, and almost guaranteed to be more accurate. The thing about python/ruby/perl is that when you do things the programmatic way you get safety by default. You don't have to remember to 'find -0 | xargs -0' and thing aren't going to bite you in the butt when filenames have spaces in them.

Yes, you can make all that work with shell (heaven knows I've done it), but it's more work and sadly isn't shell's default state.

To add, if you need/want to you can write python and perl inline in bash like so:

  python3 - <<'END'
  print('hello world')
  END

The advantage is that you still have only one shell script to ship, which makes sense when the python code inside it is specific to that one script.

Remember to use the <<'END' variant so variable expansion isn't done inside the heredoc. On the other hand, $ doesn't do anything in python and if you're feeling like it (please make sure everybody else is also feeling like it) you can use variable substitution there, in effect generating a python script and executing it.

Playing around with it, I see that using <<'END' does not expand $variables in the block, but <<END does expand them. My google-fu is failing me; what's happening here?

It's two different syntaxes. Like the difference between '$VAR' (no expansion, literal string) and "$VAR" (with expansion). See

http://tldp.org/LDP/abs/html/quoting.html

http://tldp.org/LDP/abs/html/here-docs.html

I generally, whatever I'm writing, start with bash or considering bash. Usually that means I start with bash, excluding the obvious cases such as graphics or doing heavy data processing where I know from start that line-based approach isn't enough.

Then, when bash isn't enough, I write the difficult parts in Python as standalone utilities and use those from bash. Consequently, I already have a library of little tools in my ~/bin that are generic enough so that I can just use them from bash right away. But every now and then I need to write a special tool that my bash program can use.

Most of my programs "finish" after this stage and they can live for years as a completed bash script calling helper programs. They do their job and there's nothing to add. As long as the program continues to "live on" I just make changes to the bash script or the helper programs.

If at some point I need features that bash can't offer, such as more speed or more complex data processing, I begin to consider rewriting my program in one language. But this is only after the first revision of the program is already complete and I know exactly what to do when porting.

Thus, porting the program is initially only about rewriting in another language, not developing it further at the same time. The first months or years of using the bash script turn into a prototyping stage, and this allows my rewrite be concise, well-thought and clear-partitioned.

In other words, the rewrite becomes a new stable baseline that contains only the best of my bash prototyping, and only solves a perfectly scoped problem. As soon as the port becomes a drop-in replacement for my original bash program I switch to using it and start a development branch to support the new features, if any. Usually the newly gained speed is enough already.

And usually the rewrite happens using C or Scheme. There's rarely enough point in rewriting the script in Python which may already be used to some extent in utility programs that the script calls. At that point I don't want to extend the program but lift it onto a new platform to cut some old baggage that's not needed anymore. There's a certain kind of joy rewriting in a new language something that you know completely before hand. This allows you to only work with the language since the underlying problem is already solved.

Coincidentally, this new C or Scheme program might then, one day in the future, be called from a new bash script...

Do you have any examples of a tool/workflow that's made its way from bash to bash+python to C?

God yes. I spent a year maintaining ~90k lines of bash. Interesting experience...

up-voted just to ease the pain

Thanks, man.

Actually, it was kinda neat for the first 3 months or so, to have reason to dive deep and flesh out the dark corners of my bash knowledge. There's a few things I picked up that have served me well since - and I'd lived in the shell for years beforehand... The balance wasn't kinda neat.

I'd go as far as saying "having more than a dozen lines". Bash syntax for anything other than basic execution and piping a few commands, is a disaster IMHO. Anything that requires logic should be written in a "real" scripting language.

Python is my choice when I need a bit of logic in my scripts, but it's just a personal choice of course. A good addition for simplifying execution is the "sh" library that lets you call system commands easily as if they were python functions: http://amoffat.github.io/sh/

Agreed; I would also say that you should run from Bash not when you need anything more than a simple array, but when you need any kind of array, period.

The advice on functions also gives me pause because that helps you write longer Bash scripts...not a good idea. Functions are of limited use because they can't really return values (just an exit status.) If a function has to return something, it generally prints it to stdout...which requires use of command substitution to get the result...nasty.

Shells are good for interactive use and for starting other programs, but it's a lot easier to take a full-featured language and make it work like a shell than it is to take a shell and make it behave like a full-featured language.

I'd take this blog post, add considerably to the "when you shouldn't use Bash" list, and put that at the top of the blog post, and then say "but if you absolutely must use Bash for something that is not trivial, here are some things that can make the process manageably nasty instead of truly horrid."

I love the very last list »Signs that you should not be using a bash script«. That should be a required part of every language/tool introduction/tutorial.

So very often people lose track of when to use what tools. (Although admittedly, so very often people are forced into some tools by external constraints.)

I found this web page, from the author of musl libc, very insightful:

http://www.etalabs.net/sh_tricks.html

Shell scripts are great, I use and write them every day (and quite advanced ones, too). But it's very hard to make a shell script robust.

Unfortunately it's hard to find a replacement that is stable and installed everywhere. Perl is pretty close. And python too, if you are careful about making your script compatible with all the different versions.

A link on HN about improving bash and it wasn't instructions on how to install zsh. I'm pleasantly surprised :)

I like bash and all, but a well tuned zsh is amazing. I was hesitant for a while but it really improved my workflow in the shell.

I switched from bash to zsh some times ago, and miss it when it's not there.. but TBH, the only real difference for me is zsh can turn 'cd s/l/f/p[tab]' into 'cd some/long/file/path' - all other features I'm aware of either exist in Bash too, or aren't useful for me.

What are some of the differences that made such a difference for you?

off the top of my head:

- tab completion on a shit-ton of stuff (configure scripts, makefiles, ssh hosts, aptitude packages, git branches)

- partially entered command? push up to look at similar commands you have previously entered

- immediate propagation of zsh_history to all open shells

you could (correctly) argue that you can do all that in bash. of course, shell scripting is turing complete, so yes, you can. but with zsh, it's already there, and ready.

tab completion is a default feature of GNU BASH

it tab completes more than bash does. switches, parameters, servers defined in your .ssh/config file etc. it's really impressive. coupled with the jump/mark script[1] it's amazing.

mark . marks the current directory in a folder full of symlinks.

jump <tab> expands to a list of your favorite folders.

super simple and extremely low overhead.

[1] http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-f...

Well, bash tab-completes with arbitrary complexity, seeing as it can be set to call functions... I assume the same is true of zsh, so the real question has to be adjusted for particular setup. There are a number of interesting questions of that form. Off the top of my head: The minimal setup I get from simplest packages in distro X at time Y. The maximal setup I can extract from distro X at time Y. The typical install I am likely to sit down at if it is not my own. The setup I am likely to evolve to after working with the system for it for time T...

Bash has wonderful completion, all the stuff you listed. It's in the "bash-completion" package, and usually just an "apt-get install bash-completion" away.

I use bash's reverse incremental search ("C-r") to do that. Is the zsh that much better than that?

The history sounds neat, except that I tend to segregate my terminal tabs to specific tasks/projects so I'm not very likely to want a command from one tab in another. Also I always set my bash history file to /dev/null so I don't have to worry about leaving sensitive data lying around, which means I new tabs are always a clean slate.

> Bash has wonderful completion, all the stuff you listed. It's in the "bash-completion" package, and usually just an "apt-get install bash-completion" away.

i don't doubt that at all - but it's hardly default bash, and i frequently don't have root on the systems i use (whereas zsh is usually there)

> I use bash's reverse incremental search ("C-r") to do that. Is the zsh that much better than that?

hands down yes. nobody had to explain how "up-complete" (i don't know what's it's called, but i'm calling it that because that's what it does) works to me. simple and intuitive!

> Also I always set my bash history file to /dev/null

fair enough - different people have different uses for things, i thoroughly recommend you try zsh though, because, why not! have a poke around the excellent oh-my-zsh as well, try out some themes.

i thought i had a good work flow with bash, and that there was really any room for improvement. after a few hours of zsh on my laptop, it was my shell everywhere :)

Also, M-q to stash the current command, run another command, and have the first command pasted back to the prompt, with point in the right location.

That's neat, you could almost convert me.

For Bash, M-# (alt-shift-3) puts a comment token '#' at the start of the current line and begins a new empty line. Handy if you are 150 characters deep in a command and realise you need to do something else for a moment (i.e. momentarily in the correct sense). You can store your 150 char command and come back to it, it is in the history and can be retrieved with a couple of Up-arrows.

  cd s*/l*/f*/p*

Yep!

And thanks to oh-my-zsh (https://github.com/robbyrussell/oh-my-zsh) it really requires very little fiddling, maybe 10 minutes worth.

I'm a recent zsh + oh-my-zsh convert, and I have a severe case of "why didn't I do this before"s.

Here is a great gallery of oh-my-zsh themes: http://zshthem.es/all/

Yea, it's pretty awesome.

What's really neat is you can even use it on Windows.

My personal setup is Cygwin Zsh inside Console2 (tabbed cmd.exe shell) proxied via ansicon.exe (makes cmd.exe ansi color escape aware). It's not quite as nice as say Konsole on linux or iTerm2 on a mac, but it's the best I've found for windows.

Setup guide:

Setup console2 like this http://www.hanselman.com/blog/Console2ABetterWindowsCommandP...

This post sets up ansicon: http://www.kevwebdev.com/blog/in-search-of-a-better-windows-...

I'm stuck on Windows at work, and landed on ConEmu as my terminal emulator (running bash, though - I've never made the jump to zsh):

https://code.google.com/p/conemu-maximus5/

I haven't tried Console2, so I can't provide a comparison, but here's Scott Hanselman deciding to switch: http://www.hanselman.com/blog/ConEmuTheWindowsTerminalConsol...

See also antigen(https://github.com/zsh-users/antigen), it works with oh-my-zsh and makes adding extra plugins a lot cleaner.

I keep switching back and forth between the two. The latest versions of bash have recovered a lot of ground, and zsh (in my experience) has a tendency to get really slow. Maybe it's the command completion here, or the history search there, but it just tries to be too sophisticated for my purposes (huge cluster environment with countless executables in my PATH, which by the way is constantly changing)

The thing about a shell (much like an editor, really) is that if you invest in it too far it becomes much more frustrating than reasonable to go without it. If you tend to shell in to a lot of servers that you lack control over, getting used to zsh niceties just makes the rest of your life that much harder. At least in my experience.

What if you invested in learning a shell that was installed on all servers you lack control over?

What if you invested in learning POSIX sh?

Wouldn't your scripts and one liners written in POSIX sh work in zsh, bash, etc.?

You could be comfortable on any server, no matter what the shell.

Even servers that default to tcsh still have /bin/sh.

I'm not saying don't do that, and I'm comfortable using and scripting for anything from POSIX sh to zsh, can get by in fish, etc, etc.

But if you're going to dive deep on any of them, bash has a pretty unique and strong argument for being quite likely to be available on most of the machines you work with on a day to day basis.

By "you" do you mean me, personally?

The servers I use do not have bash installed by default.

The only time I use it is when I am forced to by an installation script.

I install bash on an as needed basis, and when I'm done using it, I uninstall it. I just have no need for it otherwise.

What is the shell on Android? Is it bash?

No, definitely meant it as a general thing. And really, directed towards people working with servers, and not to people working with embedded devices (like Android) where keeping to POSIX (or specifically ash, which is what android and busybox use) is probably a necessity.

Many servers don't have bash installed as part of their base distribution, but if they're going to have another shell already installed it's much more likely to be bash than zsh.

  complete -r

disables Bash 'smart tab completion', which in theory is a great idea ( use tab to complete arguments or only list files applicable to the program ) but which never seems to work properly for me.

Disabling it saves a lot of frustrated tab-banging.

Something is terribly wrong with your setup if command line completion is not working.

A lot of installs come with over-complete smart completion configurations that make <tab> take several seconds (or even 10s of seconds) to complete in fairly common situations.

Please give me one or two examples of "fairly common situations" where it takes >=20 seconds for bash to respond to the tab.

Filename completion on remote systems using ssh/scp. It is a fairly common situation for me, although I wouldn’t say it takes 20s. Maybe 2-5s?

Setting up SSH multiplexing with ControlPersist[1] can help quite a bit here, since after the first connection you don't have to go through the init/connection phase for subsequent completions.

You can even preemptively fire up a master connection to commonly accessed hosts to avoid the initial delay.

[1] https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing

Ah, that’s interesting! I assumed that something like that was going on, as subsequent completions take much shorter (in the same command), but I did not know about preemptively connecting to common hosts.

Thank you very much, I’ll try to play around with it :)

it regularly happens when I want to expand ~username to /home/username

If It takes 20 seconds for bash to convert `~` to `/home/` something is terribly wrong with your system.

I'm guessing either they are using nis (which it often takes 20s for a passwd lookup) or they have done ~foo/<Tab> which needs to list the home directory; large home directorys on networks can easily take 20s for list.

Neither of those is what I was talking about though, as they aren't really part of the smart complete. To answer your question, I don't have an example, I just know that I tried ubuntu a while ago and smart complete would hang all the time. I don't currently have an ubuntu machine, so I can't try to track it down.

When you are working on a networked filesystem, and someone else has foobared it by doing something stupid.

I've had this, but could never be bothered to track down exactly what was causing it (but one thing was ubuntu looking up unknown commands using apt-cache).

I think I "solved" it by ensuring I'd correctly typed the start of what I wanted before hitting tab. Which is better and faster anyway. It fixed me...

Ubuntu's `command-not-found` is terrible but it does not affect bash-completion. CNF runs after you have told bash to execute a command. Tab completion is before you tell bash to run the command.

Well it sure did in mine! Probably a different config from your version (10.04 netbook).

BTW: Ahem. Good sir, I know what tab completion is. I've even added to it for some of my scripts (nothing fancy though).

Nice here document feature I have found recently is heredoc with pipe, e.g.

  cat <<REQUEST_BODY |      
  {
    "from" : 0,
    "size" : 40
  }
  REQUEST_BODY
  curl http://localhost -d @-

It allows to pass heredoc text to standard input of next command.

This is a prime example of useless use of cat. Heredoc already means "pass this as stdin", there's no need to pipe it. Your example without cat:

    curl http://localhost -d @- <<REQUEST_BODY
    {
      "from" : 0,
      "size" : 40
    }
    REQUEST_BODY

Well, maybe in that case it is. But I like to separate the commands so that the data flow looks sequential. That way it makes more sense to me.

Much as I hate the cargocult invocation of "UUOC", this is actually nicer because it reads more sanely - the HEREDOC is properly linked to the curl whereas the previous has the curl seemingly adrift on its own line unattached to the HEREDOC.

I like modularity:

    request_body() {
        cat <<REQUEST_BODY |      
        {
            "from" : 0,
            "size" : 40
        }
        REQUEST_BODY
    }

    get_url() {
        curl http://localhost -d @-
    }

    request_body | get_url

I find it helps readability when you come back to it a year later. Of course, it's also easy to parameterize the body, if needed.

/readability sometimes trumps YAGNI

My issue with this is that unless you're maintaining a lot of context and understanding the precise weirdness of HEREDOC piping, that looks at first glance like you're catting the HEREDOC to STDOUT and then running a random curl command. It's clever, sure, but it's harder to read and maintain.

You can also write curl after the pipe

    cat <<REQUEST_BODY | curl http://localhost -d @-
      {
        "from" : 0,
        "size" : 40
      }
    REQUEST_BODY

are we going to get a better bash at one point? I've always felt like the only thing bash scripts are good at describing is I/O redirection. But conditionals, dealing with variables, pretty much everything else is frustrating and error-prone

I use fish as my main shell and its slightly better, but just testing things on variables can be a huge mess.

I'm making shok (http://shok.io) as an alternative, but it's in its infancy (read: currently useless). The basics should be usable in a few months.

shok proposes that the solution to awkward-shell-syntax is to syntactically separate the programming language from program-invocation.

Well, we've consistently gotten a better bash as bash development has proceeded. As for something that removes the warts and doesn't cut out anything too important and winds up with adoption - it'll be interesting to see...

I enjoy using fish as my main shell, but as soon as I ran into a "curl oneliner install" that failed in fish (in my case Homebrew's at the bottom of http://brew.sh/) and required me to jump into bash. I enjoy fish so much I continue to use it, but I am in a state of fear that I will at some point encounter some failing shell script that leaves my system in a broken state.

Do you have any recommendations to make fish play better with shell scripts intended for bash?

In fish, type "bash" then return. Now paste in the curl oneliner and hit return. Finally, type "exit" to exit bash and go back to fish.

I used fish for about 6 months or so and loved it - with this exception. Initially I'd just hop into bash to do whatever I needed (as another comment suggests), but what was the "last straw" for me was not being to have some of the convenience functions that virtualenvwrapper exposes for working with python virtualenvs. My solution was to switch to zsh/oh-my-zsh - I'm pretty happy with it so far, although I do miss a few things from fish (namely the auto suggest/complete when typing previous commands).

First, don't make fish your system shell. Instead have whatever terminal emulator you use launch it by default. Second feel free to jump back to bash for running things that are bash dependent.

I was jumping into bash a lot working with ROS until I made my own ROS fish bindings and still go back when I'm copying and pasting, e.g., code for grabbing some commit from Gerrit.

I'd really recommend using `set -x` or `bash -x script` to sanity check all the commands and expected output.

See http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03...

The article mentioned both -x and -v, didn't it? ;)

Try moving all bash code into functions leaving only global variable/constant definitions and a call to “main” at the top-level.

One of my main complaints with bash.. the file is evaluated in order - you can't call a function on the line before it's declared.

This fails:

    #!/usr/bin/env bash
    bar='bar'
    foofunction

    foofunction(){
      echo 'foo'
      echo $bar
    }

Basically you have to write your entire script in reverse, and i'm unaware of a good way to get around it.

C is the same, but at least with C it's possible to declare functions before providing definitions. Is it possible that it's the same in bash?

I really hate writing things in reverse, but a decent style in Bash is to put everything in functions, w/ a main() at the top:

    #! /usr/bin/env bash
    set -e -u
    bar='bar'
    
    main() {
      foofunction
    }
    
    foofunction() {
      echo 'foo'
      echo $bar
    }

    main "$@"

Well, what would you do instead? It's an interpreted language. When you type "foofunction" at a prompt, you don't want it to wait around in case you define that to have meaning later.

You probably want the interpreter to be smart: "Am I loading a script from a file, or am I receiving instructions interactively on the command-line?" But now there's two modes of execution, and code in a bash script won't work if you type it into the prompt yourself. That's a bit uncomfortable.

If you'd like to see a decently written piece of bash that incorporates many of these suggestions, check out pass, the standard unix password manager.

Project page: http://www.zx2c4.com/projects/password-store/ Source: http://git.zx2c4.com/password-store/tree/src/password-store....

Excellent, bash the good parts. More than 15 minutes though.

Googling bashlint, shlint turns up some discussion (bash -n, ksh -n, zsh -n, some github projects), but I doubt they cover this article's specifics - though most (all?) of it could be automatically checked. I think some could be automatically added (e.g. set -o nounset) - perhaps a bash-subset (or coffeescript-style language) possible...

I have had good results with Syntastic for Vim. Running

    :SyntasticInfo

lists 'sh' as the linting program for bash scripts.

Try out shellcheck - there's an online checker at http://www.shellcheck.net/, and if you like it, the source for it is on github at https://github.com/koalaman/shellcheck.

The author uses ${} a lot more than I see in most code. Is it helpful to always use the ${var} syntax instead of simply writing $var?

I can see universal application of ${} being advantageous in avoiding accidental "$foo_bar where you meant ${foo}_bar" situations, and ${} makes it clearer that you're referencing a variable. The only cost would seem to be more typing.

I also like http://google-styleguide.googlecode.com/svn/trunk/shell.xml but of course some things that work well for Google might not work for you.

>> This will take care of two very common errors: >> Referencing undefined variables (which default to "") >> Ignoring failing commands

Better is subjective... About half my scripts depend on those features. For default arguments, and fail early.

He's advocating fail early with "set -e". And as for the other, you can allow overrides with syntax like:

    COULD_BE_SET=${COULD_BE_SET:altvalue}

There is no need for long set flags, e.g. use

  set -e

and not:

  set -o err

etc.

They are functionally equivalent, but the longhand versions are clearly more readable/greppable/google-able. The longhand versions have the exact same benefits as calling a variable eg. `target_file` instead of `a`.

Usually true but not always. Calling a for loop iterator "index" instead of "i" just adds unnecessary noise, for example.

I think "set -e" in bash should become common on the same level, it's pretty rare that you really want a script to continue after an unguarded error return.

Yes, and you can always set +e if you need to undo it.

I do, however, really despise that you use - to turn these things on and + to turn them off. Without the + you just think of it as being like command line flags, but then you run into +e and the whole thing goes topsy turvy.

I call my for loop variables idx. I never use single letter variable names, though I won't call someone out for it in a small block. Being able to highlight a variable in my editor, even when it's only used in a 5 line block of code is useful. Highlighting all letter i's is not.

\<i\>

It's a lot easier for my fingers to type /idx than /\<i\> and it also easier to scan for idx than i. I'm not saying that everyone should do as I do but I find it frustrating dealing with single letter variables personally.

I agree /\<i\> is a slight pain. Often the easiest way to get there is getting the cursor on one of the i's and hitting *.

What kind of editor are you using?! Any decent ide will give you syntax aware highlighting of selected variable, not word, and even most simple text editors highlight words separated by spaces, commas etc. I've never seen an editor that highlights the exact selection. I have actually missed that feature once or twice but for coding the other behavior is usually much more preferable.

I just use vim.

I set both those options in every script i write. So i do it like this:

  #! /bin/bash -eu

Because i do it in every script, readability and greppability are not important to me; i just need to apply the flags and get on with the script. Taking up two whole lines for them just adds noise.

If i was more selective in my use of those flags, then i would agree that the long forms were preferable, for the reasons given.

causing your script to run incorrectly as `bash script`.

Does bash check the flags in the shebang if you run it with bash instead of directly?

One drawback with this is that if you, or someone, does

  % bash script.sh

to run your script, then the shebang line will never be seen, and your script will run with -e off. If the "set -e" is explicitly given, this won't happen.

As you can guess, I've done this by mistake. One case is after transferring or unarchiving files where execute flags get turned off by mistake. Or using utilities, like job schedulers, that are tricky in whether they run the script as an executable, or via a shell interpreter.

Bash scripting is one of those domains that people just randomly enter without a full walk through the documentation. It's one of those things that almost everybody learns by copy and paste.

Given that, I think it's nice to be a bit more explicit, as it means the person who reads your script to copy and paste a bit out of it is more likely to learn something new in the process.

Not that the copy and paste approach to learning a language is good, but that's how these things typically go in practice.

I think "set -e" may be more idiomatic. While not scientific, there are more instances (by about an order of magnitude) of "set -e" than "set -o errexit" found via a code search on GitHub.

Anecdotally, I've nearly always seen "set -e" in bash scripts I've worked with over the years (if the option is there at all).

Agreed. It's like

. ./file

versus

source ./file

Try googling for that if your Google foo is weak. "shell script dot"? "bash dot"? (edit: it seems Google search has gotten smarter or probably has an improved profile for me since this time it actually points to somewhat useful stuff; last time I searched for this was 8 or so years ago)

I had problems getting an upstart script to recognise 'source' - it had to be . ./foo. From memory, 'source' is a bashism.

I would argue they make the code less readable, merely because they are not the original names. I certainly had no idea what "set -o errexit" did—I've never ever seen it in the wild and it doesn't sound familiar to me at all (and I've done a lot of shell scripting).

But "set -e"? Oh yeah, I use that all the time. It also has the benefit of being the same name as the command line option. That's especially useful for "set -x" which I use all the time when debugging bash, especially in the form "bash -x myscript".

I would never say "tar --extract --file" either–it's just not done.

My shell scripts almost always contain "set -e -x", and I always forget which is exit on error, and which is echo commands. The long form would help me avoid that problem.

"The two settings also have shorthands (“-u” and “-e”) but the longer versions are more readable."

One thing I've liked is throwing ${PIPESTATUS[*]} at the front of my PS1.

Or you can just use Zsh, which is superior in any way to Bash. ;)

Except existing deployment spread :)

Reply on news.ycombinator.com